[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 07:58:09.193791 7576 test_util.cc:268] Using random seed: -154715347
I20240502 07:58:09.200070 7576 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:58:09.200120 7576 ts_itest-base.cc:113] --------------
I20240502 07:58:09.200130 7576 ts_itest-base.cc:114] 6 tablet servers
I20240502 07:58:09.200140 7576 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:58:09.200150 7576 ts_itest-base.cc:116] --------------
2024-05-02T07:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:58:09Z Disabled control of system clock
I20240502 07:58:09.208840 7576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPB91DV/build/debug/bin/kudu
/tmp/dist-test-taskPB91DV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.102.62:33557
--webserver_interface=127.7.102.62
--webserver_port=0
--builtin_ntp_servers=127.7.102.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.102.62:33557
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=40 with env {}
I20240502 07:58:09.363319 7576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPB91DV/build/debug/bin/kudu as pid 7588
I20240502 07:58:09.363452 7576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:58:09.421121 7576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPB91DV/build/debug/bin/kudu
/tmp/dist-test-taskPB91DV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.102.1:0
--local_ip_for_outbound_sockets=127.7.102.1
--webserver_interface=127.7.102.1
--webserver_port=0
--tserver_master_addrs=127.7.102.62:33557
--builtin_ntp_servers=127.7.102.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 07:58:09.586092 7576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPB91DV/build/debug/bin/kudu as pid 7679
I20240502 07:58:09.586211 7576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:58:09.588351 7576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPB91DV/build/debug/bin/kudu
/tmp/dist-test-taskPB91DV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.102.2:0
--local_ip_for_outbound_sockets=127.7.102.2
--webserver_interface=127.7.102.2
--webserver_port=0
--tserver_master_addrs=127.7.102.62:33557
--builtin_ntp_servers=127.7.102.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 07:58:09.753098 7576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPB91DV/build/debug/bin/kudu as pid 7810
I20240502 07:58:09.753196 7576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:58:09.755070 7576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPB91DV/build/debug/bin/kudu
/tmp/dist-test-taskPB91DV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.102.3:0
--local_ip_for_outbound_sockets=127.7.102.3
--webserver_interface=127.7.102.3
--webserver_port=0
--tserver_master_addrs=127.7.102.62:33557
--builtin_ntp_servers=127.7.102.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 07:58:09.909291 7576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPB91DV/build/debug/bin/kudu as pid 7941
I20240502 07:58:09.909399 7576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:58:09.911481 7576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPB91DV/build/debug/bin/kudu
/tmp/dist-test-taskPB91DV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.102.4:0
--local_ip_for_outbound_sockets=127.7.102.4
--webserver_interface=127.7.102.4
--webserver_port=0
--tserver_master_addrs=127.7.102.62:33557
--builtin_ntp_servers=127.7.102.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 07:58:10.075805 7576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPB91DV/build/debug/bin/kudu as pid 8072
I20240502 07:58:10.075920 7576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:58:10.077786 7576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPB91DV/build/debug/bin/kudu
/tmp/dist-test-taskPB91DV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.102.5:0
--local_ip_for_outbound_sockets=127.7.102.5
--webserver_interface=127.7.102.5
--webserver_port=0
--tserver_master_addrs=127.7.102.62:33557
--builtin_ntp_servers=127.7.102.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 07:58:10.232087 7576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPB91DV/build/debug/bin/kudu as pid 8203
I20240502 07:58:10.232192 7576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:58:10.234323 7576 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPB91DV/build/debug/bin/kudu
/tmp/dist-test-taskPB91DV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.102.6:0
--local_ip_for_outbound_sockets=127.7.102.6
--webserver_interface=127.7.102.6
--webserver_port=0
--tserver_master_addrs=127.7.102.62:33557
--builtin_ntp_servers=127.7.102.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.25
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=1
--consensus_rpc_timeout_ms=1000
--heartbeat_interval_ms=250
--raft_heartbeat_interval_ms=100 with env {}
I20240502 07:58:10.409466 7576 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPB91DV/build/debug/bin/kudu as pid 8334
I20240502 07:58:10.409565 7576 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 07:58:10.412148 7576 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 07:58:10.482406 7576 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 07:58:10.483820 7576 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3431b1bd79314960b65c682dfbac6faf to finish bootstrapping
I20240502 07:58:10.489097 7576 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0d49a1872bfb44438ddd33147425394b to finish bootstrapping
I20240502 07:58:10.493353 7576 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0989b8a20bc84920b78ff5715057eccd to finish bootstrapping
I20240502 07:58:10.498101 7576 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 46ea7f77369547a4a3c2efde3dbdaef2 to finish bootstrapping
I20240502 07:58:10.502128 7576 ts_itest-base.cc:233] Waiting for 0 tablets on tserver de3d6ddb9f9f45fb90580be4d3a29134 to finish bootstrapping
I20240502 07:58:10.506795 7576 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 46790388ed664987be699606caa521b3 to finish bootstrapping
I20240502 07:58:10.510573 7576 test_util.cc:268] Using random seed: -153398550
F20240502 07:58:20.933503 9152 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-10c55c91bc7e40d3a8c34689a7093d13.queue_time_us, tablet-download-pool-10c55c91bc7e40d3a8c34689a7093d13.queue_time_us) (tablet-download-pool-10c55c91bc7e40d3a8c34689a7093d13.run_cpu_time_us, tablet-download-pool-10c55c91bc7e40d3a8c34689a7093d13.run_cpu_time_us) (tablet-download-pool-10c55c91bc7e40d3a8c34689a7093d13.run_wall_time_us, tablet-download-pool-10c55c91bc7e40d3a8c34689a7093d13.run_wall_time_us) (tablet-download-pool-14ca4fcb8fb644dbac369993562f78a7.queue_time_us, tablet-download-pool-14ca4fcb8fb644dbac369993562f78a7.queue_time_us) (tablet-download-pool-14ca4fcb8fb644dbac369993562f78a7.run_cpu_time_us, tablet-download-pool-14ca4fcb8fb644dbac369993562f78a7.run_cpu_time_us) (tablet-download-pool-14ca4fcb8fb644dbac369993562f78a7.run_wall_time_us, tablet-download-pool-14ca4fcb8fb644dbac369993562f78a7.run_wall_time_us) (tablet-download-pool-38837defa02f46af869539ca27935acf.queue_time_us, tablet-download-pool-38837defa02f46af869539ca27935acf.queue_time_us) (tablet-download-pool-38837defa02f46af869539ca27935acf.run_cpu_time_us, tablet-download-pool-38837defa02f46af869539ca27935acf.run_cpu_time_us) (tablet-download-pool-38837defa02f46af869539ca27935acf.run_wall_time_us, tablet-download-pool-38837defa02f46af869539ca27935acf.run_wall_time_us) (tablet-download-pool-3e3c7485ce844096842aa704f5e57d99.queue_time_us, tablet-download-pool-3e3c7485ce844096842aa704f5e57d99.queue_time_us) (tablet-download-pool-3e3c7485ce844096842aa704f5e57d99.run_cpu_time_us, tablet-download-pool-3e3c7485ce844096842aa704f5e57d99.run_cpu_time_us) (tablet-download-pool-3e3c7485ce844096842aa704f5e57d99.run_wall_time_us, tablet-download-pool-3e3c7485ce844096842aa704f5e57d99.run_wall_time_us) (tablet-download-pool-43a0132f18964c11934d2ae92fa6fd23.queue_time_us, tablet-download-pool-43a0132f18964c11934d2ae92fa6fd23.queue_time_us) (tablet-download-pool-43a0132f18964c11934d2ae92fa6fd23.run_cpu_time_us, tablet-download-pool-43a0132f18964c11934d2ae92fa6fd23.run_cpu_time_us) (tablet-download-pool-43a0132f18964c11934d2ae92fa6fd23.run_wall_time_us, tablet-download-pool-43a0132f18964c11934d2ae92fa6fd23.run_wall_time_us) (tablet-download-pool-487e5bd717614edf84cb94ab25650e0c.queue_time_us, tablet-download-pool-487e5bd717614edf84cb94ab25650e0c.queue_time_us) (tablet-download-pool-87402f8e99b6494c9b04126d3e80a531.queue_time_us, tablet-download-pool-87402f8e99b6494c9b04126d3e80a531.queue_time_us) (tablet-download-pool-87402f8e99b6494c9b04126d3e80a531.run_cpu_time_us, tablet-download-pool-87402f8e99b6494c9b04126d3e80a531.run_cpu_time_us) (tablet-download-pool-87402f8e99b6494c9b04126d3e80a531.run_wall_time_us, tablet-download-pool-87402f8e99b6494c9b04126d3e80a531.run_wall_time_us) (tablet-download-pool-8afebc5667c84641ba3a916a7ff4d68d.queue_time_us, tablet-download-pool-8afebc5667c84641ba3a916a7ff4d68d.queue_time_us) (tablet-download-pool-8afebc5667c84641ba3a916a7ff4d68d.run_cpu_time_us, tablet-download-pool-8afebc5667c84641ba3a916a7ff4d68d.run_cpu_time_us) (tablet-download-pool-8afebc5667c84641ba3a916a7ff4d68d.run_wall_time_us, tablet-download-pool-8afebc5667c84641ba3a916a7ff4d68d.run_wall_time_us) (tablet-download-pool-920a9542a1e041ea9c6265ab6a7167dd.queue_time_us, tablet-download-pool-920a9542a1e041ea9c6265ab6a7167dd.queue_time_us) (tablet-download-pool-920a9542a1e041ea9c6265ab6a7167dd.run_cpu_time_us, tablet-download-pool-920a9542a1e041ea9c6265ab6a7167dd.run_cpu_time_us) (tablet-download-pool-920a9542a1e041ea9c6265ab6a7167dd.run_wall_time_us, tablet-download-pool-920a9542a1e041ea9c6265ab6a7167dd.run_wall_time_us) (tablet-download-pool-9904156126074dd3a1b67acbd3c8d30e.queue_time_us, tablet-download-pool-9904156126074dd3a1b67acbd3c8d30e.queue_time_us) (tablet-download-pool-9904156126074dd3a1b67acbd3c8d30e.run_cpu_time_us, tablet-download-pool-9904156126074dd3a1b67acbd3c8d30e.run_cpu_time_us) (tablet-download-pool-9904156126074dd3a1b67acbd3c8d30e.run_wall_time_us, tablet-download-pool-9904156126074dd3a1b67acbd3c8d30e.run_wall_time_us) (tablet-download-pool-bf84605ce0ff4106b46a729059f1144a.queue_time_us, tablet-download-pool-bf84605ce0ff4106b46a729059f1144a.queue_time_us) (tablet-download-pool-bf84605ce0ff4106b46a729059f1144a.run_cpu_time_us, tablet-download-pool-bf84605ce0ff4106b46a729059f1144a.run_cpu_time_us) (tablet-download-pool-bf84605ce0ff4106b46a729059f1144a.run_wall_time_us, tablet-download-pool-bf84605ce0ff4106b46a729059f1144a.run_wall_time_us) (tablet-download-pool-c4ca699d57e44e7980249feea9e75a08.queue_time_us, tablet-download-pool-c4ca699d57e44e7980249feea9e75a08.queue_time_us) (tablet-download-pool-c4ca699d57e44e7980249feea9e75a08.run_cpu_time_us, tablet-download-pool-c4ca699d57e44e7980249feea9e75a08.run_cpu_time_us) (tablet-download-pool-c4ca699d57e44e7980249feea9e75a08.run_wall_time_us, tablet-download-pool-c4ca699d57e44e7980249feea9e75a08.run_wall_time_us) (tablet-download-pool-d5de0020a16f4e34a9ce868f03f4d011.queue_time_us, tablet-download-pool-d5de0020a16f4e34a9ce868f03f4d011.queue_time_us) (tablet-download-pool-d5de0020a16f4e34a9ce868f03f4d011.run_cpu_time_us, tablet-download-pool-d5de0020a16f4e34a9ce868f03f4d011.run_cpu_time_us) (tablet-download-pool-d5de0020a16f4e34a9ce868f03f4d011.run_wall_time_us, tablet-download-pool-d5de0020a16f4e34a9ce868f03f4d011.run_wall_time_us) (tablet-download-pool-dcdd2542346246b986628bc28c7128af.queue_time_us, tablet-download-pool-dcdd2542346246b986628bc28c7128af.queue_time_us) (tablet-download-pool-dcdd2542346246b986628bc28c7128af.run_cpu_time_us, tablet-download-pool-dcdd2542346246b986628bc28c7128af.run_cpu_time_us) (tablet-download-pool-dcdd2542346246b986628bc28c7128af.run_wall_time_us, tablet-download-pool-dcdd2542346246b986628bc28c7128af.run_wall_time_us) (tablet-download-pool-ea3cc024783e44539ea2d16d80764191.queue_time_us, tablet-download-pool-ea3cc024783e44539ea2d16d80764191.queue_time_us) (tablet-download-pool-ea3cc024783e44539ea2d16d80764191.run_cpu_time_us, tablet-download-pool-ea3cc024783e44539ea2d16d80764191.run_cpu_time_us) (tablet-download-pool-ea3cc024783e44539ea2d16d80764191.run_wall_time_us, tablet-download-pool-ea3cc024783e44539ea2d16d80764191.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f40622d6dcd google::LogMessage::Fail() at ??:0
@ 0x7f40622dab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f40622d67cc google::LogMessage::Flush() at ??:0
@ 0x7f40622d7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4063740044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f4063724f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f4063722efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f40662d577a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f40662dea45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f406635f76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f40663734f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f406635e226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f406636dc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c20411775c std::function<>::operator()() at ??:0
@ 0x7f4063727e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f40637286ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f406372a03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c20411775c std::function<>::operator()() at ??:0
@ 0x7f4063717807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4063f556db start_thread at ??:0
@ 0x7f406148d71f clone at ??:0
W20240502 07:58:21.553179 8497 meta_cache.cc:302] tablet 172eca85e69e4bb4b49e1235602d9ff9: replica 46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833) has failed: Network error: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111)
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 6 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | HEALTHY | <none> | 0 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
local_ip_for_outbound_sockets | 127.7.102.1 | experimental | 127.7.102.1:34873
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.7.102.1:34873
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.1:34873, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 3 other server(s)
Tablet Summary
Tablet 21154fde3dad4c23a1406908cd00aef1 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 46ea7f77369547a4a3c2efde3dbdaef2
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 9e47348ba6084666a9d69230ee09a1cb of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 0989b8a20bc84920b78ff5715057eccd
C = 0d49a1872bfb44438ddd33147425394b
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A B C* D~ | 2 | 62 | Yes
B | A* B C | 1 | 59 | Yes
C | A* B C | 1 | 59 | Yes
Tablet fae7043faa3b47cabc9b9fa4ce602912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 46ea7f77369547a4a3c2efde3dbdaef2
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 920a9542a1e041ea9c6265ab6a7167dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 60 | Yes
B | A B* C | 1 | 60 | Yes
C | [config not available] | | |
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 3e3c7485ce844096842aa704f5e57d99 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 70 | Yes
B | A* B C | 2 | 70 | Yes
C | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet ea3cc024783e44539ea2d16d80764191 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 69 | Yes
B | A* B C | 1 | 69 | Yes
C | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | -1 | Yes
C | A B* C | 2 | -1 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 10c55c91bc7e40d3a8c34689a7093d13 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 58 | Yes
B | A* B C | 1 | 58 | Yes
C | [config not available] | | |
Tablet 5018283bdc72492b94af5ed2aaa23df0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 53 | Yes
C | A B C* | 2 | -1 | Yes
Tablet 38837defa02f46af869539ca27935acf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 62 | Yes
B | A* B C | 1 | 62 | Yes
C | [config not available] | | |
Tablet 27996efbff1e4463a805002def7c8ad6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 57 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 3ddf4edff3ea47d5872d9507b3c4e42c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 68 | Yes
C | A B C* | 3 | -1 | Yes
Tablet 036bdf0a845f4519bb8dccc2a9cc530a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 62 | Yes
C | A B C* | 1 | -1 | Yes
Tablet b005bec2ec6547f289b67a384d7f8f2f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 56 | Yes
C | A B C* | 2 | -1 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | -1 | Yes
C | A B* C | 3 | -1 | Yes
Tablet 87402f8e99b6494c9b04126d3e80a531 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 65 | Yes
B | A* B C | 3 | 65 | Yes
C | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | -1 | Yes
C | A B* C | 2 | -1 | Yes
Tablet 0bf6db8ab865477db44c64a075c18aa5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 62 | Yes
C | A B C* | 3 | -1 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet c4ca699d57e44e7980249feea9e75a08 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 68 | Yes
B | A* B C | 2 | 68 | Yes
C | [config not available] | | |
Tablet 5cd30ae6e481440ba02e6662915dd0d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 59 | Yes
C | A B C* | 2 | -1 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 403b80c5d3394445b513ca9c6caa3990 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 60 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | CONSENSUS_MISMATCH | 40 | 12 | 0 | 27 | 1
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 20
Median | 21
Third Quartile | 22
Maximum | 28
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 28
Small | 3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 123
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
F20240502 07:58:24.998999 9679 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-013d45d6e24b4471972e109ed960a9b2.queue_time_us, tablet-download-pool-013d45d6e24b4471972e109ed960a9b2.queue_time_us) (tablet-download-pool-013d45d6e24b4471972e109ed960a9b2.run_cpu_time_us, tablet-download-pool-013d45d6e24b4471972e109ed960a9b2.run_cpu_time_us) (tablet-download-pool-013d45d6e24b4471972e109ed960a9b2.run_wall_time_us, tablet-download-pool-013d45d6e24b4471972e109ed960a9b2.run_wall_time_us) (tablet-download-pool-036bdf0a845f4519bb8dccc2a9cc530a.queue_time_us, tablet-download-pool-036bdf0a845f4519bb8dccc2a9cc530a.queue_time_us) (tablet-download-pool-036bdf0a845f4519bb8dccc2a9cc530a.run_cpu_time_us, tablet-download-pool-036bdf0a845f4519bb8dccc2a9cc530a.run_cpu_time_us) (tablet-download-pool-036bdf0a845f4519bb8dccc2a9cc530a.run_wall_time_us, tablet-download-pool-036bdf0a845f4519bb8dccc2a9cc530a.run_wall_time_us) (tablet-download-pool-0bf6db8ab865477db44c64a075c18aa5.queue_time_us, tablet-download-pool-0bf6db8ab865477db44c64a075c18aa5.queue_time_us) (tablet-download-pool-0bf6db8ab865477db44c64a075c18aa5.run_cpu_time_us, tablet-download-pool-0bf6db8ab865477db44c64a075c18aa5.run_cpu_time_us) (tablet-download-pool-0bf6db8ab865477db44c64a075c18aa5.run_wall_time_us, tablet-download-pool-0bf6db8ab865477db44c64a075c18aa5.run_wall_time_us) (tablet-download-pool-21154fde3dad4c23a1406908cd00aef1.queue_time_us, tablet-download-pool-21154fde3dad4c23a1406908cd00aef1.queue_time_us) (tablet-download-pool-21154fde3dad4c23a1406908cd00aef1.run_cpu_time_us, tablet-download-pool-21154fde3dad4c23a1406908cd00aef1.run_cpu_time_us) (tablet-download-pool-21154fde3dad4c23a1406908cd00aef1.run_wall_time_us, tablet-download-pool-21154fde3dad4c23a1406908cd00aef1.run_wall_time_us) (tablet-download-pool-27996efbff1e4463a805002def7c8ad6.queue_time_us, tablet-download-pool-27996efbff1e4463a805002def7c8ad6.queue_time_us) (tablet-download-pool-27996efbff1e4463a805002def7c8ad6.run_cpu_time_us, tablet-download-pool-27996efbff1e4463a805002def7c8ad6.run_cpu_time_us) (tablet-download-pool-27996efbff1e4463a805002def7c8ad6.run_wall_time_us, tablet-download-pool-27996efbff1e4463a805002def7c8ad6.run_wall_time_us) (tablet-download-pool-3ddf4edff3ea47d5872d9507b3c4e42c.queue_time_us, tablet-download-pool-3ddf4edff3ea47d5872d9507b3c4e42c.queue_time_us) (tablet-download-pool-3ddf4edff3ea47d5872d9507b3c4e42c.run_cpu_time_us, tablet-download-pool-3ddf4edff3ea47d5872d9507b3c4e42c.run_cpu_time_us) (tablet-download-pool-3ddf4edff3ea47d5872d9507b3c4e42c.run_wall_time_us, tablet-download-pool-3ddf4edff3ea47d5872d9507b3c4e42c.run_wall_time_us) (tablet-download-pool-403b80c5d3394445b513ca9c6caa3990.queue_time_us, tablet-download-pool-403b80c5d3394445b513ca9c6caa3990.queue_time_us) (tablet-download-pool-403b80c5d3394445b513ca9c6caa3990.run_cpu_time_us, tablet-download-pool-403b80c5d3394445b513ca9c6caa3990.run_cpu_time_us) (tablet-download-pool-403b80c5d3394445b513ca9c6caa3990.run_wall_time_us, tablet-download-pool-403b80c5d3394445b513ca9c6caa3990.run_wall_time_us) (tablet-download-pool-4797ce4c6795477ea12b8f46e650ca67.queue_time_us, tablet-download-pool-4797ce4c6795477ea12b8f46e650ca67.queue_time_us) (tablet-download-pool-4797ce4c6795477ea12b8f46e650ca67.run_cpu_time_us, tablet-download-pool-4797ce4c6795477ea12b8f46e650ca67.run_cpu_time_us) (tablet-download-pool-4797ce4c6795477ea12b8f46e650ca67.run_wall_time_us, tablet-download-pool-4797ce4c6795477ea12b8f46e650ca67.run_wall_time_us) (tablet-download-pool-5018283bdc72492b94af5ed2aaa23df0.queue_time_us, tablet-download-pool-5018283bdc72492b94af5ed2aaa23df0.queue_time_us) (tablet-download-pool-5018283bdc72492b94af5ed2aaa23df0.run_cpu_time_us, tablet-download-pool-5018283bdc72492b94af5ed2aaa23df0.run_cpu_time_us) (tablet-download-pool-5018283bdc72492b94af5ed2aaa23df0.run_wall_time_us, tablet-download-pool-5018283bdc72492b94af5ed2aaa23df0.run_wall_time_us) (tablet-download-pool-5cd30ae6e481440ba02e6662915dd0d6.queue_time_us, tablet-download-pool-5cd30ae6e481440ba02e6662915dd0d6.queue_time_us) (tablet-download-pool-5cd30ae6e481440ba02e6662915dd0d6.run_cpu_time_us, tablet-download-pool-5cd30ae6e481440ba02e6662915dd0d6.run_cpu_time_us) (tablet-download-pool-5cd30ae6e481440ba02e6662915dd0d6.run_wall_time_us, tablet-download-pool-5cd30ae6e481440ba02e6662915dd0d6.run_wall_time_us) (tablet-download-pool-94ceca07f29d4673af5974febbed353d.queue_time_us, tablet-download-pool-94ceca07f29d4673af5974febbed353d.queue_time_us) (tablet-download-pool-94ceca07f29d4673af5974febbed353d.run_cpu_time_us, tablet-download-pool-94ceca07f29d4673af5974febbed353d.run_cpu_time_us) (tablet-download-pool-94ceca07f29d4673af5974febbed353d.run_wall_time_us, tablet-download-pool-94ceca07f29d4673af5974febbed353d.run_wall_time_us) (tablet-download-pool-b005bec2ec6547f289b67a384d7f8f2f.queue_time_us, tablet-download-pool-b005bec2ec6547f289b67a384d7f8f2f.queue_time_us) (tablet-download-pool-b005bec2ec6547f289b67a384d7f8f2f.run_cpu_time_us, tablet-download-pool-b005bec2ec6547f289b67a384d7f8f2f.run_cpu_time_us) (tablet-download-pool-b005bec2ec6547f289b67a384d7f8f2f.run_wall_time_us, tablet-download-pool-b005bec2ec6547f289b67a384d7f8f2f.run_wall_time_us) (tablet-download-pool-be2077381f724f22aeaf7ef6001c05f1.queue_time_us, tablet-download-pool-be2077381f724f22aeaf7ef6001c05f1.queue_time_us) (tablet-download-pool-be2077381f724f22aeaf7ef6001c05f1.run_cpu_time_us, tablet-download-pool-be2077381f724f22aeaf7ef6001c05f1.run_cpu_time_us) (tablet-download-pool-be2077381f724f22aeaf7ef6001c05f1.run_wall_time_us, tablet-download-pool-be2077381f724f22aeaf7ef6001c05f1.run_wall_time_us) (tablet-download-pool-f56fb49b4a3f4342bb47ec2e1b487505.queue_time_us, tablet-download-pool-f56fb49b4a3f4342bb47ec2e1b487505.queue_time_us) (tablet-download-pool-fae7043faa3b47cabc9b9fa4ce602912.queue_time_us, tablet-download-pool-fae7043faa3b47cabc9b9fa4ce602912.queue_time_us) (tablet-download-pool-fae7043faa3b47cabc9b9fa4ce602912.run_cpu_time_us, tablet-download-pool-fae7043faa3b47cabc9b9fa4ce602912.run_cpu_time_us) (tablet-download-pool-fae7043faa3b47cabc9b9fa4ce602912.run_wall_time_us, tablet-download-pool-fae7043faa3b47cabc9b9fa4ce602912.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f3d741e5dcd google::LogMessage::Fail() at ??:0
@ 0x7f3d741e9b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3d741e57cc google::LogMessage::Flush() at ??:0
@ 0x7f3d741e6f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3d7564f044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f3d75633f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f3d75631efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f3d781e477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f3d781eda45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f3d7826e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f3d782824f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f3d7826d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f3d7827cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561137db375c std::function<>::operator()() at ??:0
@ 0x7f3d75636e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3d756376ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3d7563903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561137db375c std::function<>::operator()() at ??:0
@ 0x7f3d75626807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3d75e646db start_thread at ??:0
@ 0x7f3d7339c71f clone at ??:0
W20240502 07:58:25.144421 8497 meta_cache.cc:302] tablet 0bf6db8ab865477db44c64a075c18aa5: replica 46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833) has failed: Network error: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111)
W20240502 07:58:25.309962 8497 connection.cc:535] client connection to 127.7.102.1:34873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet fae7043faa3b47cabc9b9fa4ce602912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 63 | Yes
B | A* B C | 1 | 63 | Yes
C | [config not available] | | |
Tablet 920a9542a1e041ea9c6265ab6a7167dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [NONVOTER]
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
D = de3d6ddb9f9f45fb90580be4d3a29134
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 1 | 65 | Yes
B | A B* C D~ | 1 | 65 | Yes
C | [config not available] | | |
D | A B* C D~ | 1 | 65 | Yes
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 61 | Yes
C | A B* C D~ | 1 | 61 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 58 | Yes
C | A B* C D~ | 1 | 58 | Yes
D | [config not available] | | |
Tablet ea3cc024783e44539ea2d16d80764191 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [NONVOTER]
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
D = de3d6ddb9f9f45fb90580be4d3a29134
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B D | 1 | 77 | Yes
B | A* B D | 1 | 77 | Yes
C | [config not available] | | |
D | A* B D | 1 | 77 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 67 | Yes
C | A B* C D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 59 | Yes
C | A B C* | 1 | 59 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 62 | Yes
C | A B* C | 5 | 62 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 66 | Yes
C | A B* C D~ | 1 | 66 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 62 | Yes
C | A B* C | 3 | 62 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 68 | Yes
C | A B* C D~ | 1 | 68 | Yes
D | [config not available] | | |
Tablet 10c55c91bc7e40d3a8c34689a7093d13 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [NONVOTER]
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
D = de3d6ddb9f9f45fb90580be4d3a29134
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 63 | Yes
B | A* B C D~ | 1 | 63 | Yes
C | [config not available] | | |
D | A* B C D~ | 1 | 63 | Yes
Tablet 5018283bdc72492b94af5ed2aaa23df0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 55 | Yes
B | A B* C | 2 | 55 | Yes
C | [config not available] | | |
Tablet 38837defa02f46af869539ca27935acf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [NONVOTER]
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 0989b8a20bc84920b78ff5715057eccd
C = 46ea7f77369547a4a3c2efde3dbdaef2
D = de3d6ddb9f9f45fb90580be4d3a29134
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 65 | Yes
B | A* B C D~ | 1 | 65 | Yes
C | [config not available] | | |
D | A B C D~ | 1 | 65 | Yes
Tablet 27996efbff1e4463a805002def7c8ad6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 59 | Yes
B | A B* C | 1 | 59 | Yes
C | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 63 | Yes
C | A B* C | 2 | 63 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 60 | Yes
C | A B* C D~ | 1 | 60 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 68 | Yes
C | A B C* | 3 | 68 | Yes
Tablet 3ddf4edff3ea47d5872d9507b3c4e42c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 70 | Yes
B | A B* C | 3 | 70 | Yes
C | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 64 | Yes
C | A B C* | 1 | 64 | Yes
Tablet 036bdf0a845f4519bb8dccc2a9cc530a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 64 | Yes
B | A B* C | 1 | 64 | Yes
C | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 66 | Yes
C | A B* C | 3 | 66 | Yes
Tablet b005bec2ec6547f289b67a384d7f8f2f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 58 | Yes
B | A B* C | 2 | 58 | Yes
C | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 67 | Yes
C | A B C* | 3 | 67 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 59 | Yes
C | A B* C D~ | 3 | 59 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 60 | Yes
C | A B* C D~ | 2 | 60 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 60 | Yes
C | A B C* | 2 | 60 | Yes
Tablet 0bf6db8ab865477db44c64a075c18aa5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 64 | Yes
B | A B* C | 3 | 64 | Yes
C | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 64 | Yes
C | A B C* | 2 | 64 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 62 | Yes
C | A B* C D~ | 1 | 62 | Yes
D | [config not available] | | |
Tablet 5cd30ae6e481440ba02e6662915dd0d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 61 | Yes
B | A B* C | 2 | 61 | Yes
C | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 57 | Yes
C | A B* C | 1 | 57 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 62 | Yes
C | A B* C D~ | 1 | 62 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 64 | Yes
C | A B* C | 4 | 64 | Yes
Tablet 403b80c5d3394445b513ca9c6caa3990 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 62 | Yes
B | A B* C | 1 | 62 | Yes
C | [config not available] | | |
Tablet 21154fde3dad4c23a1406908cd00aef1 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 4 | Yes
B | A B* C | 1 | 4 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 4 | 0 | 36 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 15
First Quartile | 20
Median | 22
Third Quartile | 27
Maximum | 33
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 138
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet 21154fde3dad4c23a1406908cd00aef1 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet 21154fde3dad4c23a1406908cd00aef1: Tablet copy aborted
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 1 | 5 | Yes
B | A B* C D~ | 1 | 5 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet fae7043faa3b47cabc9b9fa4ce602912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): not running [NONVOTER]
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap complete.
All reported replicas are:
A = 0d49a1872bfb44438ddd33147425394b
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 68 | Yes
B | A* B C D~ | 1 | 68 | Yes
C | [config not available] | | |
D | A B C D~ | 1 | 68 | Yes
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 65 | Yes
C | A B* C | 1 | 65 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 61 | Yes
C | A B* C | 1 | 61 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 70 | Yes
C | A B* C | 2 | 70 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 63 | Yes
C | A B C* D~ | 1 | 63 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 67 | Yes
C | A B* C D~ | 5 | 67 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 67 | Yes
C | A B* C D~ | 3 | 67 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 5018283bdc72492b94af5ed2aaa23df0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [NONVOTER]
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* D | 2 | 62 | Yes
B | A B* D | 2 | 62 | No
C | [config not available] | | |
D | A B* C D | 2 | 61 | Yes
Tablet 27996efbff1e4463a805002def7c8ad6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [NONVOTER]
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 1 | 62 | Yes
B | A B* C D~ | 1 | 62 | Yes
C | [config not available] | | |
D | A B* C D~ | 1 | 62 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 67 | Yes
C | A B* C D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 64 | Yes
C | A B* C | 1 | 64 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 73 | Yes
C | A B C* D~ | 3 | 73 | Yes
D | [config not available] | | |
Tablet 3ddf4edff3ea47d5872d9507b3c4e42c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [NONVOTER]
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 3 | 74 | Yes
B | A B* C D~ | 3 | 74 | Yes
C | [config not available] | | |
D | A B C D~ | 3 | 74 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 67 | Yes
C | A B C* D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 036bdf0a845f4519bb8dccc2a9cc530a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet 036bdf0a845f4519bb8dccc2a9cc530a: Tablet copy aborted
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 1 | 68 | Yes
B | A B* C D~ | 1 | 68 | Yes
C | [config not available] | | |
D | A B C D~ | 1 | 68 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 71 | Yes
C | A B* C D~ | 3 | 71 | Yes
D | [config not available] | | |
Tablet b005bec2ec6547f289b67a384d7f8f2f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet b005bec2ec6547f289b67a384d7f8f2f: Tablet copy aborted
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 2 | 62 | Yes
B | A B* C D~ | 2 | 62 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 70 | Yes
C | A B C* D~ | 3 | 70 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 61 | Yes
C | A B* C | 3 | 61 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 63 | Yes
C | A B* C | 2 | 63 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 64 | Yes
C | A B C* D~ | 2 | 64 | Yes
D | [config not available] | | |
Tablet 0bf6db8ab865477db44c64a075c18aa5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet 0bf6db8ab865477db44c64a075c18aa5: Tablet copy aborted
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 3 | 68 | Yes
B | A B* C D~ | 3 | 68 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 67 | Yes
C | A B C* D~ | 2 | 67 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 66 | Yes
C | A B* C | 1 | 66 | Yes
Tablet 5cd30ae6e481440ba02e6662915dd0d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [NONVOTER]
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* D | 2 | 66 | Yes
B | A B* D | 2 | 66 | No
C | [config not available] | | |
D | A B* D | 2 | 66 | No
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 59 | Yes
C | A B* C D~ | 1 | 59 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 65 | Yes
C | A B* C | 1 | 65 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 67 | Yes
C | A B* C D~ | 4 | 67 | Yes
D | [config not available] | | |
Tablet 403b80c5d3394445b513ca9c6caa3990 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [NONVOTER]
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 1 | 64 | Yes
B | A B* C D~ | 1 | 64 | Yes
C | [config not available] | | |
D | A B* C D~ | 1 | 64 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 8 | 0 | 32 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 21
First Quartile | 22
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 145
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.5:41583, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 64 | Yes
C | A B* C D~ | 1 | 64 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C D~ | 1 | 73 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 69 | Yes
C | A B* C | 5 | 69 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 69 | Yes
C | A B* C | 3 | 69 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 69 | Yes
C | A B* C | 2 | 69 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 75 | Yes
C | A B C* | 3 | 75 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 67 | Yes
C | A B C* D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 74 | Yes
C | A B* C | 3 | 74 | Yes
D | [config not available] | | |
Tablet b005bec2ec6547f289b67a384d7f8f2f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING [LEADER]
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [NONVOTER]
All reported replicas are:
A = de3d6ddb9f9f45fb90580be4d3a29134
B = 46790388ed664987be699606caa521b3
C = 3431b1bd79314960b65c682dfbac6faf
D = 0d49a1872bfb44438ddd33147425394b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* D | 2 | 66 | Yes
B | A B* D | 2 | 66 | Yes
C | [config not available] | | |
D | A B* D | 2 | 66 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 73 | Yes
C | A B C* | 3 | 73 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 65 | Yes
C | A B* C D~ | 3 | 65 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 66 | Yes
C | A B* C D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 67 | Yes
C | A B C* | 2 | 67 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 70 | Yes
C | A B C* | 2 | 70 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 69 | Yes
C | A B* C D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 68 | Yes
C | A B* C D~ | 1 | 68 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 68 | Yes
C | A B* C | 4 | 68 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 16 | 0 | 24 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 19
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 142
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 64 | Yes
C | A B* C D~ | 1 | 64 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 73 | Yes
C | A B* C D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 69 | Yes
C | A B* C | 5 | 69 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 69 | Yes
C | A B* C | 3 | 69 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 69 | Yes
C | A B* C | 2 | 69 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 75 | Yes
C | A B C* | 3 | 75 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 74 | Yes
C | A B* C | 3 | 74 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 73 | Yes
C | A B C* | 3 | 73 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 65 | Yes
C | A B* C D~ | 3 | 65 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 66 | Yes
C | A B* C D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 67 | Yes
C | A B C* | 2 | 67 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 70 | Yes
C | A B C* | 2 | 70 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 69 | Yes
C | A B* C D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 68 | Yes
C | A B* C D~ | 1 | 68 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 68 | Yes
C | A B* C | 4 | 68 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 64 | Yes
C | A B* C D~ | 1 | 64 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 66 | Yes
C | A B C* | 1 | 66 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 73 | Yes
C | A B* C D~ | 1 | 73 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 69 | Yes
C | A B* C | 5 | 69 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 69 | Yes
C | A B* C | 3 | 69 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 69 | Yes
C | A B* C | 2 | 69 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 75 | Yes
C | A B C* | 3 | 75 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 69 | Yes
C | A B C* | 1 | 69 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 74 | Yes
C | A B* C | 3 | 74 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 73 | Yes
C | A B C* | 3 | 73 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 70 | Yes
C | A B* C | 3 | 70 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 66 | Yes
C | A B* C D~ | 2 | 66 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 67 | Yes
C | A B C* | 2 | 67 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 70 | Yes
C | A B C* | 2 | 70 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 69 | Yes
C | A B* C D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 62 | Yes
C | A B* C | 1 | 62 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 68 | Yes
C | A B* C D~ | 1 | 68 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 68 | Yes
C | A B* C | 4 | 68 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 69 | Yes
C | A B C* D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 78 | Yes
C | A B* C | 1 | 78 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 74 | Yes
C | A B* C D~ | 5 | 74 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 74 | Yes
C | A B* C D~ | 3 | 74 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 78 | Yes
C | A B C* D~ | 3 | 78 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 77 | Yes
C | A B* C D~ | 3 | 77 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 75 | Yes
C | A B C* D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 70 | Yes
C | A B* C | 3 | 70 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 70 | Yes
C | A B* C | 2 | 70 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 73 | Yes
C | A B C* D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 72 | Yes
C | A B C* D~ | 2 | 72 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 72 | Yes
C | A B* C D~ | 4 | 72 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 67 | Yes
C | A B* C | 1 | 67 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 69 | Yes
C | A B C* D~ | 1 | 69 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 78 | Yes
C | A B* C | 1 | 78 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 74 | Yes
C | A B* C D~ | 5 | 74 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 74 | Yes
C | A B* C D~ | 3 | 74 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 73 | Yes
C | A B* C D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 78 | Yes
C | A B C* D~ | 3 | 78 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 77 | Yes
C | A B* C D~ | 3 | 77 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 75 | Yes
C | A B C* D~ | 3 | 75 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 70 | Yes
C | A B* C | 3 | 70 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 70 | Yes
C | A B* C | 2 | 70 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 73 | Yes
C | A B C* D~ | 2 | 73 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 72 | Yes
C | A B C* D~ | 2 | 72 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 71 | Yes
C | A B* C | 1 | 71 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 67 | Yes
C | A B* C D~ | 1 | 67 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 72 | Yes
C | A B* C D~ | 4 | 72 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 70 | Yes
C | A B* C | 1 | 70 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 73 | Yes
C | A B C* | 1 | 73 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 78 | Yes
C | A B* C | 1 | 78 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 79 | Yes
C | A B* C | 5 | 79 | No
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 78 | Yes
C | A B* C | 3 | 78 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 77 | Yes
C | A B* C | 1 | 77 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 71 | Yes
C | A B C* D~ | 1 | 71 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 84 | Yes
C | A B* C | 3 | 84 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 80 | Yes
C | A B C* | 3 | 80 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 76 | Yes
C | A B* C D~ | 3 | 76 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 78 | Yes
C | A B C* | 2 | 78 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 75 | Yes
C | A B* C | 4 | 75 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 17
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 140
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 73 | Yes
C | A B C* | 1 | 73 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 79 | Yes
C | A B* C | 5 | 79 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 78 | Yes
C | A B* C | 3 | 78 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 76 | Yes
C | A B C* | 1 | 76 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 84 | Yes
C | A B* C | 3 | 84 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 80 | Yes
C | A B C* | 3 | 80 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 76 | Yes
C | A B* C D~ | 3 | 76 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 78 | Yes
C | A B C* | 2 | 78 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 75 | Yes
C | A B* C | 4 | 75 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 74 | Yes
C | A B* C D~ | 1 | 74 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 72 | Yes
C | A B* C D~ | 1 | 72 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 73 | Yes
C | A B C* | 1 | 73 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 79 | Yes
C | A B* C | 5 | 79 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 78 | Yes
C | A B* C | 3 | 78 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 85 | Yes
C | A B C* | 3 | 85 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 76 | Yes
C | A B C* | 1 | 76 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 84 | Yes
C | A B* C | 3 | 84 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 80 | Yes
C | A B C* | 3 | 80 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 76 | Yes
C | A B* C D~ | 2 | 76 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 78 | Yes
C | A B C* | 2 | 78 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 77 | Yes
C | A B C* | 2 | 77 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 76 | Yes
C | A B* C D~ | 1 | 76 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 73 | Yes
C | A B* C | 1 | 73 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 75 | Yes
C | A B* C | 4 | 75 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.2:44555, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 75 | Yes
C | A B C* D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 83 | Yes
C | A B* C D~ | 5 | 83 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 81 | Yes
C | A B* C D~ | 3 | 81 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 88 | Yes
C | A B C* D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 88 | Yes
C | A B* C D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 85 | Yes
C | A B C* D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 85 | Yes
C | A B C* D~ | 2 | 85 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 82 | Yes
C | A B C* D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 83 | Yes
C | A B* C | 1 | 83 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 82 | Yes
C | A B* C D~ | 4 | 82 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 76 | Yes
C | A B* C | 1 | 76 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 87 | Yes
C | A B* C | 2 | 87 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 75 | Yes
C | A B C* D~ | 1 | 75 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 83 | Yes
C | A B* C D~ | 5 | 83 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 81 | Yes
C | A B* C D~ | 3 | 81 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 88 | Yes
C | A B C* D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 88 | Yes
C | A B* C D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 85 | Yes
C | A B C* D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 81 | Yes
C | A B* C | 3 | 81 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 78 | Yes
C | A B* C | 2 | 78 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 85 | Yes
C | A B C* D~ | 2 | 85 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 82 | Yes
C | A B C* D~ | 2 | 82 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 78 | Yes
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 83 | Yes
C | A B* C | 1 | 83 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 82 | Yes
C | A B* C D~ | 4 | 82 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.5:41583, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 80 | Yes
C | A B* C | 1 | 80 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 79 | Yes
C | A B C* | 1 | 79 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 87 | Yes
C | A B* C | 1 | 87 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 83 | Yes
C | A B* C D~ | 5 | 83 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C D~ | 1 | 94 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 81 | Yes
C | A B* C D~ | 3 | 81 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 89 | Yes
C | A B* C | 2 | 89 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 88 | Yes
C | A B C* D~ | 3 | 88 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 82 | Yes
C | A B C* D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 92 | Yes
C | A B* C | 3 | 92 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 85 | Yes
C | A B* C D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 85 | Yes
C | A B C* D~ | 2 | 85 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 87 | Yes
C | A B C* | 2 | 87 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | No
C | A B* C D~ | 1 | 78 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 88 | Yes
C | A B* C D~ | 1 | 88 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 82 | Yes
C | A B* C D~ | 4 | 82 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 139
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 79 | Yes
C | A B C* | 1 | 79 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 87 | Yes
C | A B* C | 5 | 87 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 86 | Yes
C | A B* C | 3 | 86 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 89 | Yes
C | A B* C | 2 | 89 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 94 | Yes
C | A B C* | 3 | 94 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 87 | Yes
C | A B C* | 1 | 87 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 92 | Yes
C | A B* C | 3 | 92 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 85 | Yes
C | A B* C D~ | 3 | 85 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 88 | Yes
C | A B C* | 2 | 88 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 87 | Yes
C | A B C* | 2 | 87 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 88 | Yes
C | A B* C D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 87 | Yes
C | A B* C | 4 | 87 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 83 | Yes
C | A B* C D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 82 | Yes
C | A B* C D~ | 1 | 82 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 91 | Yes
C | A B* C D~ | 2 | 91 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 79 | Yes
C | A B C* | 1 | 79 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 91 | Yes
C | A B* C D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 87 | Yes
C | A B* C | 5 | 87 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 94 | Yes
C | A B* C D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 86 | Yes
C | A B* C | 3 | 86 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 89 | Yes
C | A B* C | 2 | 89 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 85 | Yes
C | A B* C D~ | 1 | 85 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 94 | Yes
C | A B C* | 3 | 94 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 87 | Yes
C | A B C* | 1 | 87 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 92 | Yes
C | A B* C | 3 | 92 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 90 | Yes
C | A B C* | 3 | 90 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 92 | Yes
C | A B* C | 3 | 92 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 83 | Yes
C | A B* C D~ | 2 | 83 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 88 | Yes
C | A B C* | 2 | 88 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 87 | Yes
C | A B C* | 2 | 87 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 84 | Yes
C | A B* C D~ | 1 | 84 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 82 | Yes
C | A B* C | 1 | 82 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 88 | Yes
C | A B* C D~ | 1 | 88 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 87 | Yes
C | A B* C | 4 | 87 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 92 | Yes
C | A B* C | 2 | 92 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 83 | Yes
C | A B C* D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 97 | Yes
C | A B* C | 1 | 97 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 92 | Yes
C | A B* C D~ | 5 | 92 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 91 | Yes
C | A B* C D~ | 3 | 91 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 94 | Yes
C | A B* C | 1 | 94 | No
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 93 | Yes
C | A B* C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 91 | Yes
C | A B* C | 1 | 91 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 98 | Yes
C | A B C* D~ | 3 | 98 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 97 | Yes
C | A B* C D~ | 3 | 97 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 94 | Yes
C | A B C* D~ | 3 | 94 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 92 | Yes
C | A B* C | 3 | 92 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 88 | Yes
C | A B* C | 2 | 88 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 93 | Yes
C | A B C* D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 90 | Yes
C | A B C* D~ | 2 | 90 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 86 | Yes
C | A B* C D~ | 1 | 86 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 92 | Yes
C | A B* C | 1 | 92 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 90 | Yes
C | A B* C D~ | 4 | 90 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.5:41583, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 85 | Yes
C | A B* C | 1 | 85 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 92 | Yes
C | A B* C | 2 | 92 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 83 | Yes
C | A B C* D~ | 1 | 83 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 97 | Yes
C | A B* C | 1 | 97 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 92 | Yes
C | A B* C D~ | 5 | 92 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 91 | Yes
C | A B* C D~ | 3 | 91 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 94 | Yes
C | A B* C | 1 | 94 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 93 | Yes
C | A B* C D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 91 | Yes
C | A B* C | 1 | 91 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 98 | Yes
C | A B C* D~ | 3 | 98 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 97 | Yes
C | A B* C D~ | 3 | 97 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 94 | Yes
C | A B C* D~ | 3 | 94 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 92 | Yes
C | A B* C | 3 | 92 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 88 | Yes
C | A B* C | 2 | 88 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 93 | Yes
C | A B C* D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 90 | Yes
C | A B C* D~ | 2 | 90 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 86 | Yes
C | A B* C D~ | 1 | 86 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 92 | Yes
C | A B* C | 1 | 92 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 90 | Yes
C | A B* C D~ | 4 | 90 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.2:44555, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 88 | Yes
C | A B* C | 1 | 88 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 99 | Yes
C | A B* C D~ | 2 | 99 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 89 | Yes
C | A B C* | 1 | 89 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 97 | Yes
C | A B* C | 1 | 97 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 92 | Yes
C | A B* C D~ | 5 | 92 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 103 | Yes
C | A B* C D~ | 1 | 103 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 91 | Yes
C | A B* C D~ | 3 | 91 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 94 | Yes
C | A B* C | 1 | 94 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 99 | Yes
C | A B* C | 2 | 99 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 98 | Yes
C | A B C* D~ | 3 | 98 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 91 | Yes
C | A B C* D~ | 1 | 91 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 102 | Yes
C | A B* C | 3 | 102 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 94 | Yes
C | A B C* D~ | 3 | 94 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 97 | Yes
C | A B* C D~ | 3 | 97 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 93 | Yes
C | A B C* D~ | 2 | 93 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 95 | Yes
C | A B C* | 2 | 95 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 92 | Yes
C | A B* C D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 90 | Yes
C | A B* C D~ | 4 | 90 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 20
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 138
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 92 | Yes
C | A B* C D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 99 | Yes
C | A B* C D~ | 2 | 99 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 89 | Yes
C | A B C* | 1 | 89 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 102 | Yes
C | A B* C D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 96 | Yes
C | A B* C | 5 | 96 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 103 | Yes
C | A B* C D~ | 1 | 103 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 94 | Yes
C | A B* C | 3 | 94 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 98 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 99 | Yes
C | A B* C | 2 | 99 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 102 | Yes
C | A B C* | 3 | 102 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 95 | Yes
C | A B C* | 1 | 95 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 102 | Yes
C | A B* C | 3 | 102 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 98 | Yes
C | A B C* | 3 | 98 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 97 | Yes
C | A B* C D~ | 3 | 97 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 98 | Yes
C | A B C* | 2 | 98 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 95 | Yes
C | A B C* | 2 | 95 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 92 | Yes
C | A B* C D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 94 | Yes
C | A B* C | 4 | 94 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.2:44555, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 92 | Yes
C | A B* C D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 90 | Yes
C | A B* C D~ | 1 | 90 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 99 | Yes
C | A B* C D~ | 2 | 99 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 89 | Yes
C | A B C* | 1 | 89 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 102 | Yes
C | A B* C D~ | 1 | 102 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 96 | Yes
C | A B* C | 5 | 96 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 103 | Yes
C | A B* C D~ | 1 | 103 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 94 | Yes
C | A B* C | 3 | 94 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 98 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 99 | Yes
C | A B* C | 2 | 99 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 100 | Yes
C | A B* C | 1 | 100 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 102 | Yes
C | A B C* | 3 | 102 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 95 | Yes
C | A B C* | 1 | 95 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 102 | Yes
C | A B* C | 3 | 102 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 98 | Yes
C | A B C* | 3 | 98 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 101 | Yes
C | A B* C | 3 | 101 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 92 | Yes
C | A B* C D~ | 2 | 92 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 98 | Yes
C | A B C* | 2 | 98 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 95 | Yes
C | A B C* | 2 | 95 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 92 | Yes
C | A B* C D~ | 1 | 92 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 90 | Yes
C | A B* C | 1 | 90 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 96 | Yes
C | A B* C D~ | 1 | 96 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 94 | Yes
C | A B* C | 4 | 94 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 93 | Yes
C | A B* C | 1 | 93 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 103 | Yes
C | A B* C | 2 | 103 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 94 | Yes
C | A B C* D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 101 | Yes
C | A B* C D~ | 5 | 101 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 106 | Yes
C | A B* C | 1 | 106 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 100 | Yes
C | A B* C D~ | 3 | 100 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 98 | Yes
C | A B* C D~ | 1 | 98 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 101 | Yes
C | A B* C D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 100 | Yes
C | A B* C | 1 | 100 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 105 | Yes
C | A B C* D~ | 3 | 105 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 101 | Yes
C | A B C* D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 105 | Yes
C | A B* C D~ | 3 | 105 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 103 | Yes
C | A B C* D~ | 3 | 103 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 101 | Yes
C | A B* C | 3 | 101 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 104 | Yes
C | A B C* D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 98 | Yes
C | A B C* D~ | 2 | 98 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 95 | Yes
C | A B* C D~ | 1 | 95 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 100 | Yes
C | A B* C D~ | 4 | 100 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 93 | Yes
C | A B* C | 1 | 93 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 103 | Yes
C | A B* C | 2 | 103 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 94 | Yes
C | A B C* D~ | 1 | 94 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 101 | Yes
C | A B* C D~ | 5 | 101 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 106 | Yes
C | A B* C | 1 | 106 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 100 | Yes
C | A B* C D~ | 3 | 100 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 102 | Yes
C | A B* C | 1 | 102 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 101 | Yes
C | A B* C D~ | 2 | 101 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 100 | Yes
C | A B* C | 1 | 100 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 105 | Yes
C | A B C* D~ | 3 | 105 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 101 | Yes
C | A B C* D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 105 | Yes
C | A B* C D~ | 3 | 105 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 103 | Yes
C | A B C* D~ | 3 | 103 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 101 | Yes
C | A B* C | 3 | 101 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 96 | Yes
C | A B* C | 2 | 96 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 104 | Yes
C | A B C* D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 98 | Yes
C | A B C* D~ | 2 | 98 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 95 | Yes
C | A B* C | 1 | 95 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 95 | Yes
C | A B* C D~ | 1 | 95 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 100 | Yes
C | A B* C D~ | 4 | 100 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 96 | Yes
C | A B* C | 1 | 96 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 108 | Yes
C | A B* C D~ | 2 | 108 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 100 | Yes
C | A B C* | 1 | 100 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 105 | Yes
C | A B* C | 5 | 105 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 105 | Yes
C | A B* C | 3 | 105 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 102 | Yes
C | A B* C | 1 | 102 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 106 | Yes
C | A B* C D~ | 1 | 106 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 110 | Yes
C | A B C* | 3 | 110 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 101 | Yes
C | A B C* D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 109 | Yes
C | A B* C | 3 | 109 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 103 | Yes
C | A B C* D~ | 3 | 103 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 105 | Yes
C | A B* C D~ | 3 | 105 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 102 | Yes
C | A B* C D~ | 2 | 102 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 104 | Yes
C | A B C* D~ | 2 | 104 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 102 | Yes
C | A B C* | 2 | 102 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 103 | Yes
C | A B* C D~ | 1 | 103 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 100 | Yes
C | A B* C D~ | 4 | 100 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 19
First Quartile | 19
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 138
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 108 | Yes
C | A B* C D~ | 2 | 108 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 100 | Yes
C | A B C* | 1 | 100 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 114 | Yes
C | A B* C D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 105 | Yes
C | A B* C | 5 | 105 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 105 | Yes
C | A B* C | 3 | 105 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 106 | Yes
C | A B* C D~ | 1 | 106 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 110 | Yes
C | A B C* | 3 | 110 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 109 | Yes
C | A B* C | 3 | 109 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 108 | Yes
C | A B C* | 3 | 108 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 105 | Yes
C | A B* C D~ | 3 | 105 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 102 | Yes
C | A B* C D~ | 2 | 102 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 109 | Yes
C | A B C* | 2 | 109 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 102 | Yes
C | A B C* | 2 | 102 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 103 | Yes
C | A B* C D~ | 1 | 103 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 104 | Yes
C | A B* C | 4 | 104 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 99 | Yes
C | A B* C D~ | 1 | 99 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 108 | Yes
C | A B* C D~ | 2 | 108 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 100 | Yes
C | A B C* | 1 | 100 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 114 | Yes
C | A B* C D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 105 | Yes
C | A B* C | 5 | 105 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 105 | Yes
C | A B* C | 3 | 105 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 110 | Yes
C | A B C* | 3 | 110 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 105 | Yes
C | A B C* | 1 | 105 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 109 | Yes
C | A B* C | 3 | 109 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 108 | Yes
C | A B C* | 3 | 108 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 108 | Yes
C | A B* C | 3 | 108 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 102 | Yes
C | A B* C D~ | 2 | 102 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 109 | Yes
C | A B C* | 2 | 109 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 101 | Yes
C | A B* C D~ | 1 | 101 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 99 | Yes
C | A B* C | 1 | 99 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 103 | Yes
C | A B* C D~ | 1 | 103 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 104 | Yes
C | A B* C | 4 | 104 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 112 | Yes
C | A B* C | 2 | 112 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 109 | Yes
C | A B* C D~ | 5 | 109 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 114 | Yes
C | A B* C | 1 | 114 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 109 | Yes
C | A B* C D~ | 3 | 109 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 107 | Yes
C | A B* C D~ | 1 | 107 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 111 | Yes
C | A B* C D~ | 2 | 111 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 113 | Yes
C | A B C* D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 110 | Yes
C | A B C* D~ | 1 | 110 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 113 | Yes
C | A B* C D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 113 | Yes
C | A B C* D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 108 | Yes
C | A B* C | 3 | 108 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 115 | Yes
C | A B C* D~ | 2 | 115 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 106 | Yes
C | A B* C | 1 | 106 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 109 | Yes
C | A B* C D~ | 4 | 109 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 14
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 137
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 104 | Yes
C | A B* C | 1 | 104 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 112 | Yes
C | A B* C | 2 | 112 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 105 | Yes
C | A B C* D~ | 1 | 105 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 109 | Yes
C | A B* C D~ | 5 | 109 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 114 | Yes
C | A B* C | 1 | 114 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 109 | Yes
C | A B* C D~ | 3 | 109 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 111 | Yes
C | A B* C D~ | 2 | 111 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 113 | Yes
C | A B C* D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 110 | Yes
C | A B C* D~ | 1 | 110 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 113 | Yes
C | A B* C D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 113 | Yes
C | A B C* D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 108 | Yes
C | A B* C | 3 | 108 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 107 | Yes
C | A B* C | 2 | 107 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 115 | Yes
C | A B C* D~ | 2 | 115 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 109 | Yes
C | A B C* D~ | 2 | 109 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 106 | Yes
C | A B* C | 1 | 106 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 104 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 107 | Yes
C | A B* C | 1 | 107 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 109 | Yes
C | A B* C D~ | 4 | 109 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 105 | Yes
C | A B* C | 1 | 105 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 116 | Yes
C | A B* C D~ | 2 | 116 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 109 | Yes
C | A B C* | 1 | 109 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 109 | Yes
C | A B* C D~ | 5 | 109 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 110 | Yes
C | A B* C | 3 | 110 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 111 | Yes
C | A B* C | 1 | 111 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 119 | Yes
C | A B C* | 3 | 119 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 118 | Yes
C | A B* C | 3 | 118 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 113 | Yes
C | A B C* D~ | 3 | 113 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 112 | Yes
C | A B* C D~ | 3 | 112 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 115 | Yes
C | A B C* D~ | 2 | 115 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 111 | Yes
C | A B C* | 2 | 111 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C D~ | 1 | 104 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 109 | Yes
C | A B* C D~ | 4 | 109 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 140
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 116 | Yes
C | A B* C D~ | 2 | 116 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 109 | Yes
C | A B C* | 1 | 109 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 115 | Yes
C | A B* C | 5 | 115 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 110 | Yes
C | A B* C | 3 | 110 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 116 | Yes
C | A B* C D~ | 1 | 116 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 115 | Yes
C | A B* C D~ | 1 | 115 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 119 | Yes
C | A B C* | 3 | 119 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 118 | Yes
C | A B* C | 3 | 118 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 118 | Yes
C | A B C* | 3 | 118 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 112 | Yes
C | A B* C D~ | 3 | 112 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 120 | Yes
C | A B C* | 2 | 120 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 111 | Yes
C | A B C* | 2 | 111 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 114 | Yes
C | A B* C | 4 | 114 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 109 | Yes
C | A B* C D~ | 1 | 109 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 111 | Yes
C | A B* C D~ | 1 | 111 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 116 | Yes
C | A B* C D~ | 2 | 116 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 109 | Yes
C | A B C* | 1 | 109 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 115 | Yes
C | A B* C | 5 | 115 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 110 | Yes
C | A B* C | 3 | 110 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 116 | Yes
C | A B* C D~ | 1 | 116 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 118 | Yes
C | A B* C | 2 | 118 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 119 | Yes
C | A B C* | 3 | 119 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 115 | Yes
C | A B C* | 1 | 115 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 118 | Yes
C | A B* C | 3 | 118 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 118 | Yes
C | A B C* | 3 | 118 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 118 | Yes
C | A B* C | 3 | 118 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 112 | Yes
C | A B* C D~ | 2 | 112 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 120 | Yes
C | A B C* | 2 | 120 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 117 | No
C | A B C* D~ | 2 | 117 | No
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 109 | Yes
C | A B* C | 1 | 109 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 113 | Yes
C | A B* C D~ | 1 | 113 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 114 | Yes
C | A B* C | 4 | 114 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 120 | Yes
C | A B* C | 2 | 120 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 119 | Yes
C | A B* C D~ | 5 | 119 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 115 | Yes
C | A B* C D~ | 3 | 115 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 122 | Yes
C | A B* C D~ | 2 | 122 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 123 | Yes
C | A B C* D~ | 3 | 123 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 119 | Yes
C | A B C* D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 122 | Yes
C | A B* C D~ | 3 | 122 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 125 | Yes
C | A B C* D~ | 3 | 125 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 118 | Yes
C | A B* C | 3 | 118 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 117 | Yes
C | A B* C | 2 | 117 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 124 | Yes
C | A B C* D~ | 2 | 124 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 117 | Yes
C | A B C* D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 114 | Yes
C | A B* C D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 120 | Yes
C | A B* C D~ | 4 | 120 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 14
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 137
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 120 | Yes
C | A B* C | 2 | 120 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 114 | Yes
C | A B C* D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 119 | Yes
C | A B* C D~ | 5 | 119 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 124 | Yes
C | A B* C | 1 | 124 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 115 | Yes
C | A B* C D~ | 3 | 115 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 122 | Yes
C | A B* C D~ | 2 | 122 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 123 | Yes
C | A B C* D~ | 3 | 123 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 119 | Yes
C | A B C* D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 122 | Yes
C | A B* C D~ | 3 | 122 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 125 | Yes
C | A B C* D~ | 3 | 125 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 121 | Yes
C | A B* C D~ | 3 | 121 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 117 | Yes
C | A B* C | 2 | 117 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 124 | Yes
C | A B C* D~ | 2 | 124 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 117 | Yes
C | A B C* D~ | 2 | 117 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 114 | Yes
C | A B* C D~ | 1 | 114 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 117 | Yes
C | A B* C | 1 | 117 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 120 | Yes
C | A B* C D~ | 4 | 120 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 115 | Yes
C | A B* C | 1 | 115 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 124 | Yes
C | A B* C D~ | 2 | 124 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 117 | Yes
C | A B C* | 1 | 117 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 123 | Yes
C | A B* C | 5 | 123 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 119 | Yes
C | A B* C | 3 | 119 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 120 | Yes
C | A B* C | 1 | 120 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 126 | Yes
C | A B* C | 2 | 126 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 130 | Yes
C | A B C* | 3 | 130 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 127 | Yes
C | A B* C | 3 | 127 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 125 | Yes
C | A B C* D~ | 3 | 125 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 121 | Yes
C | A B* C D~ | 3 | 121 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | Yes
C | A B* C D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 127 | Yes
C | A B C* | 2 | 127 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 120 | Yes
C | A B C* | 2 | 120 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 116 | Yes
C | A B* C | 1 | 116 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 124 | Yes
C | A B* C | 4 | 124 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 19
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 137
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 124 | Yes
C | A B* C D~ | 2 | 124 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 117 | Yes
C | A B C* | 1 | 117 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 123 | Yes
C | A B* C | 5 | 123 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 119 | Yes
C | A B* C | 3 | 119 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 126 | Yes
C | A B* C | 2 | 126 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 130 | Yes
C | A B C* | 3 | 130 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 127 | Yes
C | A B* C | 3 | 127 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 128 | Yes
C | A B C* | 3 | 128 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 121 | Yes
C | A B* C D~ | 3 | 121 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | Yes
C | A B* C D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 127 | Yes
C | A B C* | 2 | 127 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 120 | Yes
C | A B C* | 2 | 120 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 116 | Yes
C | A B* C | 1 | 116 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 121 | Yes
C | A B* C D~ | 1 | 121 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 124 | Yes
C | A B* C | 4 | 124 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 119 | Yes
C | A B* C D~ | 1 | 119 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 124 | Yes
C | A B* C D~ | 2 | 124 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 117 | Yes
C | A B C* | 1 | 117 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 135 | Yes
C | A B* C D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 123 | Yes
C | A B* C | 5 | 123 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 119 | Yes
C | A B* C | 3 | 119 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 124 | Yes
C | A B* C D~ | 1 | 124 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 126 | Yes
C | A B* C | 2 | 126 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 130 | Yes
C | A B* C | 1 | 130 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 130 | Yes
C | A B C* | 3 | 130 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 125 | Yes
C | A B C* | 1 | 125 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 127 | Yes
C | A B* C | 3 | 127 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 128 | Yes
C | A B C* | 3 | 128 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 125 | Yes
C | A B* C | 3 | 125 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 120 | Yes
C | A B* C D~ | 2 | 120 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 127 | Yes
C | A B C* | 2 | 127 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 126 | Yes
C | A B C* D~ | 2 | 126 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 120 | Yes
C | A B* C D~ | 1 | 120 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 116 | Yes
C | A B* C | 1 | 116 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 124 | Yes
C | A B* C | 4 | 124 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.2:44555, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 123 | Yes
C | A B* C | 1 | 123 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 132 | Yes
C | A B* C D~ | 2 | 132 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 122 | Yes
C | A B C* | 1 | 122 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 130 | Yes
C | A B* C | 5 | 130 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 133 | Yes
C | A B* C D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 125 | Yes
C | A B* C | 3 | 125 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 126 | Yes
C | A B* C | 1 | 126 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 131 | Yes
C | A B* C | 2 | 131 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 137 | Yes
C | A B C* | 3 | 137 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 131 | Yes
C | A B C* | 1 | 131 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 132 | Yes
C | A B* C | 3 | 132 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 135 | Yes
C | A B C* | 3 | 135 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 126 | Yes
C | A B* C D~ | 3 | 126 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 127 | Yes
C | A B* C D~ | 2 | 127 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 127 | Yes
C | A B C* | 2 | 127 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 127 | Yes
C | A B* C D~ | 1 | 127 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 130 | Yes
C | A B* C | 4 | 130 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.2:44555, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 132 | Yes
C | A B* C D~ | 2 | 132 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 122 | Yes
C | A B C* | 1 | 122 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 130 | Yes
C | A B* C | 5 | 130 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 133 | Yes
C | A B* C D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 125 | Yes
C | A B* C | 3 | 125 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 131 | Yes
C | A B* C | 2 | 131 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 137 | Yes
C | A B C* | 3 | 137 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 131 | Yes
C | A B C* | 1 | 131 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 132 | Yes
C | A B* C | 3 | 132 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 135 | Yes
C | A B C* | 3 | 135 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 131 | Yes
C | A B* C | 3 | 131 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 127 | Yes
C | A B* C D~ | 2 | 127 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 127 | Yes
C | A B C* | 2 | 127 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 125 | Yes
C | A B* C D~ | 1 | 125 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 127 | Yes
C | A B* C D~ | 1 | 127 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 130 | Yes
C | A B* C | 4 | 130 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 134
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 128 | Yes
C | A B* C | 1 | 128 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 126 | Yes
C | A B C* D~ | 1 | 126 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 134 | Yes
C | A B* C D~ | 5 | 134 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 133 | Yes
C | A B* C D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 129 | Yes
C | A B* C D~ | 3 | 129 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 128 | Yes
C | A B* C D~ | 1 | 128 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 135 | Yes
C | A B* C D~ | 2 | 135 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 143 | Yes
C | A B C* D~ | 3 | 143 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 137 | Yes
C | A B C* D~ | 1 | 137 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 134 | Yes
C | A B* C D~ | 3 | 134 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 135 | Yes
C | A B C* | 3 | 135 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 131 | Yes
C | A B* C | 3 | 131 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 134 | Yes
C | A B C* | 2 | 134 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 131 | Yes
C | A B C* D~ | 2 | 131 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 121 | Yes
C | A B* C | 1 | 121 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 130 | Yes
C | A B* C | 4 | 130 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 20
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 132 | Yes
C | A B* C | 1 | 132 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 128 | Yes
C | A B* C | 1 | 128 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 126 | Yes
C | A B C* D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 134 | Yes
C | A B* C D~ | 5 | 134 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 129 | Yes
C | A B* C D~ | 3 | 129 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 135 | Yes
C | A B* C D~ | 2 | 135 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 143 | Yes
C | A B C* D~ | 3 | 143 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 137 | Yes
C | A B C* D~ | 1 | 137 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 134 | Yes
C | A B* C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 138 | Yes
C | A B C* D~ | 3 | 138 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 131 | Yes
C | A B* C | 3 | 131 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 137 | Yes
C | A B C* D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 131 | Yes
C | A B C* D~ | 2 | 131 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 134 | Yes
C | A B* C D~ | 4 | 134 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 132 | Yes
C | A B* C | 1 | 132 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 128 | Yes
C | A B* C | 1 | 128 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 136 | Yes
C | A B* C | 2 | 136 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 126 | Yes
C | A B C* D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 134 | Yes
C | A B* C D~ | 5 | 134 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 136 | Yes
C | A B* C | 1 | 136 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 129 | Yes
C | A B* C D~ | 3 | 129 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 135 | Yes
C | A B* C D~ | 2 | 135 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 141 | Yes
C | A B* C D~ | 1 | 141 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 143 | Yes
C | A B C* D~ | 3 | 143 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 137 | Yes
C | A B C* D~ | 1 | 137 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 134 | Yes
C | A B* C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 138 | Yes
C | A B C* D~ | 3 | 138 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 134 | Yes
C | A B* C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 132 | Yes
C | A B* C | 2 | 132 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 137 | Yes
C | A B C* D~ | 2 | 137 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 135 | Yes
C | A B C* | 2 | 135 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 129 | Yes
C | A B* C | 1 | 129 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 126 | Yes
C | A B* C D~ | 1 | 126 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 133 | Yes
C | A B* C | 1 | 133 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 134 | Yes
C | A B* C D~ | 4 | 134 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 136
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.3:42137, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 136 | Yes
C | A B* C D~ | 1 | 136 | No
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 141 | Yes
C | A B* C D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 131 | Yes
C | A B C* | 1 | 131 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 151 | Yes
C | A B* C D~ | 1 | 151 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 139 | Yes
C | A B* C | 5 | 139 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 132 | Yes
C | A B* C | 3 | 132 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 134 | Yes
C | A B* C | 1 | 134 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 140 | Yes
C | A B* C | 2 | 140 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 141 | Yes
C | A B* C D~ | 1 | 141 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 146 | Yes
C | A B C* | 3 | 146 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 140 | Yes
C | A B C* | 1 | 140 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 138 | Yes
C | A B* C | 3 | 138 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 145 | Yes
C | A B C* | 3 | 145 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 134 | Yes
C | A B* C D~ | 3 | 134 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 135 | Yes
C | A B* C D~ | 2 | 135 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 141 | Yes
C | A B C* | 2 | 141 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 135 | Yes
C | A B C* | 2 | 135 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 133 | Yes
C | A B* C D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 132 | Yes
C | A B* C | 1 | 132 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 139 | Yes
C | A B* C | 4 | 139 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.5:41583, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 136 | Yes
C | A B* C D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 131 | Yes
C | A B* C D~ | 1 | 131 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 141 | Yes
C | A B* C D~ | 2 | 141 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 131 | Yes
C | A B C* | 1 | 131 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 151 | Yes
C | A B* C D~ | 1 | 151 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 139 | Yes
C | A B* C | 5 | 139 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 132 | Yes
C | A B* C | 3 | 132 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 140 | Yes
C | A B* C | 2 | 140 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 141 | Yes
C | A B* C D~ | 1 | 141 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 146 | Yes
C | A B C* | 3 | 146 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 140 | Yes
C | A B C* | 1 | 140 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 138 | Yes
C | A B* C | 3 | 138 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 145 | Yes
C | A B C* | 3 | 145 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 140 | Yes
C | A B* C | 3 | 140 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 135 | Yes
C | A B* C D~ | 2 | 135 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 141 | Yes
C | A B C* | 2 | 141 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 135 | Yes
C | A B C* | 2 | 135 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 133 | Yes
C | A B* C D~ | 1 | 133 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 132 | Yes
C | A B* C | 1 | 132 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 139 | Yes
C | A B* C | 4 | 139 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 136 | Yes
C | A B* C D~ | 1 | 136 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | Yes
C | A B* C | 1 | 137 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 147 | Yes
C | A B* C | 2 | 147 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 135 | Yes
C | A B C* D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 151 | Yes
C | A B* C D~ | 1 | 151 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 146 | Yes
C | A B* C D~ | 5 | 146 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 137 | Yes
C | A B* C D~ | 3 | 137 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 138 | Yes
C | A B* C D~ | 1 | 138 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 145 | Yes
C | A B* C D~ | 2 | 145 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 151 | Yes
C | A B C* D~ | 3 | 151 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 143 | No
C | A B* C | 3 | 138 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 145 | Yes
C | A B C* | 3 | 145 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 140 | Yes
C | A B* C | 3 | 140 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 146 | Yes
C | A B C* D~ | 2 | 146 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 140 | Yes
C | A B C* D~ | 2 | 140 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 139 | Yes
C | A B* C | 1 | 139 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 132 | Yes
C | A B* C | 1 | 132 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 139 | Yes
C | A B* C | 4 | 139 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 16
First Quartile | 17
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 139 | Yes
C | A B* C | 1 | 139 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 137 | Yes
C | A B* C | 1 | 137 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 147 | Yes
C | A B* C | 2 | 147 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 135 | Yes
C | A B C* D~ | 1 | 135 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 146 | Yes
C | A B* C D~ | 5 | 146 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 146 | Yes
C | A B* C | 1 | 146 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 137 | Yes
C | A B* C D~ | 3 | 137 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 145 | Yes
C | A B* C D~ | 2 | 145 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 145 | Yes
C | A B* C | 1 | 145 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 151 | Yes
C | A B C* D~ | 3 | 151 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 143 | Yes
C | A B* C D~ | 3 | 143 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 148 | Yes
C | A B C* D~ | 3 | 148 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 140 | Yes
C | A B* C | 3 | 140 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 146 | Yes
C | A B C* D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 140 | Yes
C | A B C* D~ | 2 | 140 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 139 | Yes
C | A B* C | 1 | 139 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 134 | Yes
C | A B* C D~ | 1 | 134 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 145 | Yes
C | A B* C D~ | 4 | 145 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 139 | Yes
C | A B* C | 1 | 139 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 147 | Yes
C | A B* C | 2 | 147 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 139 | Yes
C | A B C* | 1 | 139 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 155 | Yes
C | A B* C | 1 | 155 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 146 | Yes
C | A B* C D~ | 5 | 146 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 146 | Yes
C | A B* C | 1 | 146 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 137 | Yes
C | A B* C D~ | 3 | 137 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 145 | Yes
C | A B* C D~ | 2 | 145 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 150 | Yes
C | A B* C D~ | 1 | 150 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 154 | No
C | A B C* | 3 | 154 | No
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 143 | Yes
C | A B* C D~ | 3 | 143 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 148 | Yes
C | A B C* D~ | 3 | 148 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 145 | Yes
C | A B* C D~ | 3 | 145 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 141 | Yes
C | A B* C | 2 | 141 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 146 | Yes
C | A B C* D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 144 | Yes
C | A B C* | 2 | 144 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 134 | Yes
C | A B* C D~ | 1 | 134 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 147 | Yes
C | A B* C D~ | 1 | 147 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 145 | Yes
C | A B* C D~ | 4 | 145 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.6:37519, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 144 | Yes
C | A B* C D~ | 1 | 144 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 151 | Yes
C | A B* C D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 139 | Yes
C | A B C* | 1 | 139 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 160 | Yes
C | A B* C D~ | 1 | 160 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 151 | Yes
C | A B* C | 5 | 151 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 151 | Yes
C | A B* C D~ | 1 | 151 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 140 | Yes
C | A B* C | 3 | 140 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 143 | Yes
C | A B* C | 1 | 143 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 149 | Yes
C | A B* C | 2 | 149 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 150 | Yes
C | A B* C D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 154 | Yes
C | A B C* | 3 | 154 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 147 | Yes
C | A B* C | 3 | 147 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 153 | Yes
C | A B C* | 3 | 153 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 145 | Yes
C | A B* C D~ | 3 | 145 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 146 | Yes
C | A B* C D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 149 | Yes
C | A B C* | 2 | 149 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 144 | Yes
C | A B C* | 2 | 144 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 139 | Yes
C | A B* C | 1 | 139 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 147 | Yes
C | A B* C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 149 | Yes
C | A B* C | 4 | 149 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.6:37519, tserver@127.7.102.2:44555, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 144 | Yes
C | A B* C D~ | 1 | 144 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 151 | Yes
C | A B* C D~ | 2 | 151 | Yes
D | [config not available] | | |
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 139 | Yes
C | A B C* | 1 | 139 | Yes
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 160 | Yes
C | A B* C D~ | 1 | 160 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 151 | Yes
C | A B* C | 5 | 151 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 151 | Yes
C | A B* C D~ | 1 | 151 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 140 | Yes
C | A B* C | 3 | 140 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 149 | Yes
C | A B* C | 2 | 149 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 150 | Yes
C | A B* C D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 154 | Yes
C | A B C* | 3 | 154 | Yes
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | 145 | Yes
C | A B C* | 1 | 145 | Yes
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 147 | Yes
C | A B* C | 3 | 147 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 153 | Yes
C | A B C* | 3 | 153 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 150 | Yes
C | A B* C | 3 | 150 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 146 | Yes
C | A B* C D~ | 2 | 146 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 149 | Yes
C | A B C* | 2 | 149 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 144 | Yes
C | A B C* | 2 | 144 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 142 | Yes
C | A B* C D~ | 1 | 142 | Yes
D | [config not available] | | |
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 139 | Yes
C | A B* C | 1 | 139 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 147 | Yes
C | A B* C D~ | 1 | 147 | Yes
D | [config not available] | | |
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 4 | 149 | Yes
C | A B* C | 4 | 149 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Small | 46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 133
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 144 | Yes
C | A B* C D~ | 1 | 144 | Yes
D | [config not available] | | |
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 156 | Yes
C | A B* C | 2 | 156 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 166 | Yes
C | A B* C | 1 | 166 | Yes
D | [config not available] | | |
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 154 | Yes
C | A B* C D~ | 5 | 154 | Yes
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 156 | Yes
C | A B* C | 1 | 156 | Yes
D | [config not available] | | |
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 142 | Yes
C | A B* C D~ | 3 | 142 | Yes
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 155 | Yes
C | A B* C D~ | 2 | 155 | Yes
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C | 1 | 154 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 159 | Yes
C | A B C* D~ | 3 | 159 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 152 | Yes
C | A B* C D~ | 3 | 152 | Yes
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 153 | Yes
C | A B C* | 3 | 153 | Yes
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 150 | Yes
C | A B* C | 3 | 150 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 151 | Yes
C | A B* C | 2 | 151 | Yes
D | [config not available] | | |
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 154 | Yes
C | A B C* D~ | 2 | 154 | Yes
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 149 | Yes
C | A B C* D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 146 | Yes
C | A B* C | 1 | 146 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 139 | Yes
C | A B* C | 1 | 139 | Yes
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 155 | No
C | A B* C | 4 | 149 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 15
First Quartile | 17
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 132
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 156 | Yes
C | A B* C | 2 | 156 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 166 | Yes
C | A B* C | 1 | 166 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 154 | Yes
C | A B* C D~ | 5 | 154 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 156 | Yes
C | A B* C | 1 | 156 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 142 | Yes
C | A B* C D~ | 3 | 142 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 155 | Yes
C | A B* C D~ | 2 | 155 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 154 | Yes
C | A B* C | 1 | 154 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 159 | Yes
C | A B C* D~ | 3 | 159 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 152 | Yes
C | A B* C D~ | 3 | 152 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 156 | Yes
C | A B C* D~ | 3 | 156 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 150 | Yes
C | A B* C | 3 | 150 | Yes
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 151 | Yes
C | A B* C | 2 | 151 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 154 | Yes
C | A B C* D~ | 2 | 154 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 149 | Yes
C | A B C* D~ | 2 | 149 | Yes
D | [config not available] | | |
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 146 | Yes
C | A B* C | 1 | 146 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 152 | Yes
C | A B* C | 1 | 152 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 155 | Yes
C | A B* C D~ | 4 | 155 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 12
First Quartile | 21
Median | 23
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
452e39cda2ef45688030b3ea58942562 | 127.7.102.62:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.102.20:44599 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
0989b8a20bc84920b78ff5715057eccd | 127.7.102.3:42137 | HEALTHY | <none> | 7 | 0
0d49a1872bfb44438ddd33147425394b | 127.7.102.2:44555 | HEALTHY | <none> | 14 | 0
46790388ed664987be699606caa521b3 | 127.7.102.6:37519 | HEALTHY | <none> | 9 | 0
de3d6ddb9f9f45fb90580be4d3a29134 | 127.7.102.5:41583 | HEALTHY | <none> | 11 | 0
3431b1bd79314960b65c682dfbac6faf | 127.7.102.1:34873 | UNAVAILABLE | <none> | n/a | n/a
46ea7f77369547a4a3c2efde3dbdaef2 | 127.7.102.4:33833 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.7.102.1:34873: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.1:34873: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.7.102.4:33833: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.7.102.4:33833: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
local_ip_for_outbound_sockets | 127.7.102.2 | experimental | 127.7.102.2:44555
local_ip_for_outbound_sockets | 127.7.102.3 | experimental | 127.7.102.3:42137
local_ip_for_outbound_sockets | 127.7.102.5 | experimental | 127.7.102.5:41583
local_ip_for_outbound_sockets | 127.7.102.6 | experimental | 127.7.102.6:37519
never_fsync | true | unsafe,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.7.102.2:44555
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.7.102.3:42137
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.7.102.5:41583
server_dump_info_path | /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.7.102.6:37519
tablet_copy_early_session_timeout_prob | 0.25 | unsafe,runtime | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.7.102.20:44599 | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
time_source | builtin | 127.7.102.3:42137, 127.7.102.2:44555, 127.7.102.6:37519, and 1 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.7.102.62:33557, tserver@127.7.102.3:42137, tserver@127.7.102.5:41583, and 2 other server(s)
Tablet Summary
Tablet be2077381f724f22aeaf7ef6001c05f1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 132609125eab44018bc8f0b0ca38ca9a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 147 | Yes
C | A B* C | 1 | 147 | Yes
Tablet 172eca85e69e4bb4b49e1235602d9ff9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 156 | Yes
C | A B* C | 2 | 156 | Yes
Tablet 14ca4fcb8fb644dbac369993562f78a7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 143 | Yes
C | A B C* D~ | 1 | 143 | Yes
D | [config not available] | | |
Tablet 94ceca07f29d4673af5974febbed353d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 166 | Yes
C | A B* C | 1 | 166 | Yes
Tablet 78ddd535f6884f2f879f126eb2b3d055 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 5 | 154 | Yes
C | A B* C D~ | 5 | 154 | Yes
D | [config not available] | | |
Tablet 013d45d6e24b4471972e109ed960a9b2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 156 | Yes
C | A B* C | 1 | 156 | Yes
Tablet 9904156126074dd3a1b67acbd3c8d30e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 142 | Yes
C | A B* C D~ | 3 | 142 | Yes
D | [config not available] | | |
Tablet 4797ce4c6795477ea12b8f46e650ca67 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 151 | Yes
C | A B* C | 1 | 151 | Yes
Tablet 16b14c0cb3914a27b9e427bfaf10745e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 2 | 155 | Yes
C | A B* C D~ | 2 | 155 | Yes
D | [config not available] | | |
Tablet 51488ad07ce246ce8fc6bac8bcc17299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 160 | Yes
C | A B* C D~ | 1 | 160 | Yes
Tablet dcdd2542346246b986628bc28c7128af of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 164 | Yes
C | A B C* | 3 | 164 | Yes
D | [config not available] | | |
Tablet 8afebc5667c84641ba3a916a7ff4d68d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 1 | 150 | Yes
C | A B C* D~ | 1 | 150 | Yes
D | [config not available] | | |
Tablet 38be6f725dfe48a4bc63d29dd134e598 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 152 | Yes
C | A B* C D~ | 3 | 152 | Yes
D | [config not available] | | |
Tablet 487e5bd717614edf84cb94ab25650e0c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 3 | 156 | Yes
C | A B C* D~ | 3 | 156 | Yes
D | [config not available] | | |
Tablet 63fb07947b1e4b4da9bb2d34a4d1278c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable [NONVOTER]
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 155 | Yes
C | A B* C D~ | 3 | 155 | Yes
D | [config not available] | | |
Tablet 40734f06ed5e4bd3b76c19a0b4b0e299 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 2 | 151 | Yes
C | A B* C | 2 | 151 | Yes
Tablet d5de0020a16f4e34a9ce868f03f4d011 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | [config not available] | | |
B | A B C* D~ | 2 | 154 | Yes
C | A B C* D~ | 2 | 154 | Yes
D | [config not available] | | |
Tablet bf84605ce0ff4106b46a729059f1144a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING [LEADER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 2 | 152 | Yes
C | A B C* | 2 | 152 | Yes
Tablet f56fb49b4a3f4342bb47ec2e1b487505 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 146 | Yes
C | A B* C | 1 | 146 | Yes
Tablet 43a0132f18964c11934d2ae92fa6fd23 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 146 | Yes
C | A B* C D~ | 1 | 146 | Yes
D | [config not available] | | |
Tablet 18e82716a3e840c08a66e4157eb9c4d2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable
de3d6ddb9f9f45fb90580be4d3a29134 (127.7.102.5:41583): RUNNING [LEADER]
46790388ed664987be699606caa521b3 (127.7.102.6:37519): RUNNING
All reported replicas are:
A = 46ea7f77369547a4a3c2efde3dbdaef2
B = de3d6ddb9f9f45fb90580be4d3a29134
C = 46790388ed664987be699606caa521b3
D = 3431b1bd79314960b65c682dfbac6faf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C D~ | 1 | 156 | Yes
C | A B* C D~ | 1 | 156 | Yes
Tablet 947ed881fd44419094d51dae9b166912 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
3431b1bd79314960b65c682dfbac6faf (127.7.102.1:34873): TS unavailable
0d49a1872bfb44438ddd33147425394b (127.7.102.2:44555): RUNNING [LEADER]
0989b8a20bc84920b78ff5715057eccd (127.7.102.3:42137): RUNNING
46ea7f77369547a4a3c2efde3dbdaef2 (127.7.102.4:33833): TS unavailable [NONVOTER]
All reported replicas are:
A = 3431b1bd79314960b65c682dfbac6faf
B = 0d49a1872bfb44438ddd33147425394b
C = 0989b8a20bc84920b78ff5715057eccd
D = 46ea7f77369547a4a3c2efde3dbdaef2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 155 | Yes
C | A B* C D~ | 4 | 155 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 40 | 17 | 0 | 23 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 13
First Quartile | 21
Median | 22
Third Quartile | 27
Maximum | 30
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 41
Replicas | 135
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
E20240502 07:58:48.195788 7576 external_mini_cluster.cc:943] Process with UUID 3431b1bd79314960b65c682dfbac6faf has crashed
E20240502 07:58:48.195885 7576 external_mini_cluster.cc:943] Process with UUID 46ea7f77369547a4a3c2efde3dbdaef2 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 07:58:48.540319 7576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPB91DV/build/debug/bin/kudu with pid 7810
I20240502 07:58:48.548380 7576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPB91DV/build/debug/bin/kudu with pid 7941
I20240502 07:58:48.554502 7576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPB91DV/build/debug/bin/kudu with pid 8203
I20240502 07:58:48.561614 7576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPB91DV/build/debug/bin/kudu with pid 8334
I20240502 07:58:48.567076 7576 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPB91DV/build/debug/bin/kudu with pid 7588
2024-05-02T07:58:48Z chronyd exiting
I20240502 07:58:48.583153 7576 test_util.cc:175] -----------------------------------------------
I20240502 07:58:48.583194 7576 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPB91DV/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714636689183421-7576-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (39391 ms)
[----------] 1 test from RaftConsensusStressITest (39391 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (39391 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240502 07:58:48.583650 7576 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~23 seconds ago