[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 20:11:45.787403 23452 test_util.cc:268] Using random seed: 411551202
I20240501 20:11:45.798389 23452 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:11:45.798465 23452 ts_itest-base.cc:113] --------------
I20240501 20:11:45.798504 23452 ts_itest-base.cc:114] 6 tablet servers
I20240501 20:11:45.798544 23452 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:11:45.798591 23452 ts_itest-base.cc:116] --------------
2024-05-01T20:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:11:45Z Disabled control of system clock
I20240501 20:11:45.821938 23452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
/tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.231.62:45525
--webserver_interface=127.22.231.62
--webserver_port=0
--builtin_ntp_servers=127.22.231.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.231.62:45525
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=20 with env {}
I20240501 20:11:46.214680 23452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu as pid 23464
I20240501 20:11:46.214955 23452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:11:46.299417 23452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
/tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.231.1:0
--local_ip_for_outbound_sockets=127.22.231.1
--webserver_interface=127.22.231.1
--webserver_port=0
--tserver_master_addrs=127.22.231.62:45525
--builtin_ntp_servers=127.22.231.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 20:11:46.745823 23452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu as pid 23554
I20240501 20:11:46.746062 23452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:11:46.749363 23452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
/tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.231.2:0
--local_ip_for_outbound_sockets=127.22.231.2
--webserver_interface=127.22.231.2
--webserver_port=0
--tserver_master_addrs=127.22.231.62:45525
--builtin_ntp_servers=127.22.231.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 20:11:47.163383 23452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu as pid 23684
I20240501 20:11:47.163573 23452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:11:47.166749 23452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
/tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.231.3:0
--local_ip_for_outbound_sockets=127.22.231.3
--webserver_interface=127.22.231.3
--webserver_port=0
--tserver_master_addrs=127.22.231.62:45525
--builtin_ntp_servers=127.22.231.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 20:11:47.590880 23452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu as pid 23815
I20240501 20:11:47.591075 23452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:11:47.594745 23452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
/tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.231.4:0
--local_ip_for_outbound_sockets=127.22.231.4
--webserver_interface=127.22.231.4
--webserver_port=0
--tserver_master_addrs=127.22.231.62:45525
--builtin_ntp_servers=127.22.231.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 20:11:48.030459 23452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu as pid 23945
I20240501 20:11:48.030659 23452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:11:48.034626 23452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
/tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.231.5:0
--local_ip_for_outbound_sockets=127.22.231.5
--webserver_interface=127.22.231.5
--webserver_port=0
--tserver_master_addrs=127.22.231.62:45525
--builtin_ntp_servers=127.22.231.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 20:11:48.470367 23452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu as pid 24075
I20240501 20:11:48.470561 23452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:11:48.473577 23452 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
/tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.231.6:0
--local_ip_for_outbound_sockets=127.22.231.6
--webserver_interface=127.22.231.6
--webserver_port=0
--tserver_master_addrs=127.22.231.62:45525
--builtin_ntp_servers=127.22.231.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 20:11:48.909107 23452 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu as pid 24205
I20240501 20:11:48.909256 23452 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:11:48.925645 23452 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 20:11:49.073577 23452 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 20:11:49.076079 23452 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 12bcc720148d426aa9ca25efe70ac6d4 to finish bootstrapping
I20240501 20:11:49.084555 23452 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c487cdc5c45c4365854a474ec10d6e54 to finish bootstrapping
I20240501 20:11:49.092703 23452 ts_itest-base.cc:233] Waiting for 1 tablets on tserver aff8dec1c558403bb403fe2e08a34305 to finish bootstrapping
I20240501 20:11:49.099174 23452 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b6298d218143477eb0770540c0cdbbcc to finish bootstrapping
I20240501 20:11:49.107185 23452 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f661a2fa62084188b9de0c0e1608bd8b to finish bootstrapping
I20240501 20:11:49.114004 23452 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b3ac395387bb4601b607ae4a8404f325 to finish bootstrapping
I20240501 20:11:49.121986 23452 test_util.cc:268] Using random seed: 414885791
W20240501 20:12:09.955835 25135 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:12:06.777111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.231.2:39099 (local address 127.0.0.1:57052)
0501 20:12:06.777569 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0501 20:12:06.777585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 20:12:06.777691 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:12:09.954406 (+3176715us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.231.2:39099: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":64}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | HEALTHY | <none> | 0 | 0
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 5 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 0 | 0
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | HEALTHY | <none> | 4 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 4 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.2:39099: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.2:39099: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.22.231.1 | experimental | 127.22.231.1:42115
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.4 | experimental | 127.22.231.4:40461
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.231.1:42115
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.231.4:40461
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.1:42115, tserver@127.22.231.3:35521, and 3 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = aff8dec1c558403bb403fe2e08a34305
C = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 4 | Yes
B | A B* C | 1 | 4 | Yes
C | [config not available] | | |
Tablet 6b64ee927588429eb0ace9f5a0241937 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | 44 | Yes
C | A* B C | 1 | 44 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
All reported replicas are:
A = aff8dec1c558403bb403fe2e08a34305
B = f661a2fa62084188b9de0c0e1608bd8b
C = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 44 | Yes
B | A* B C | 1 | 44 | Yes
C | [config not available] | | |
Tablet 1c7d48c373f349a98c71db9125837df7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 40 | Yes
C | A B* C | 1 | 40 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 3 | 42 | Yes
C | A* B C | 3 | 42 | Yes
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
All reported replicas are:
A = b6298d218143477eb0770540c0cdbbcc
B = f661a2fa62084188b9de0c0e1608bd8b
C = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 42 | Yes
B | A B* C | 2 | 42 | Yes
C | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | 43 | Yes
C | A* B C | 1 | 43 | Yes
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING [LEADER]
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
All reported replicas are:
A = b6298d218143477eb0770540c0cdbbcc
B = f661a2fa62084188b9de0c0e1608bd8b
C = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 45 | Yes
B | A* B C | 3 | 45 | Yes
C | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | 42 | Yes
C | A* B C | 1 | 42 | Yes
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
All reported replicas are:
A = b6298d218143477eb0770540c0cdbbcc
B = f661a2fa62084188b9de0c0e1608bd8b
C = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 43 | Yes
B | A B* C | 1 | 43 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 42 | Yes
C | A B* C | 1 | 42 | Yes
Tablet c6af8b3feba148dc84b2bfe7ff67807e of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING [LEADER]
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = b6298d218143477eb0770540c0cdbbcc
B = f661a2fa62084188b9de0c0e1608bd8b
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A B C | 3 | -1 | Yes
B | A B C | 3 | -1 | Yes
C | A B C | 3 | -1 | Yes
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 2 | 45 | Yes
C | A* B C | 2 | 45 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING [LEADER]
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
All reported replicas are:
A = b6298d218143477eb0770540c0cdbbcc
B = f661a2fa62084188b9de0c0e1608bd8b
C = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 43 | Yes
B | A* B C | 1 | 43 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | 43 | Yes
C | A* B C | 1 | 43 | Yes
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
All reported replicas are:
A = b6298d218143477eb0770540c0cdbbcc
B = f661a2fa62084188b9de0c0e1608bd8b
C = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 41 | Yes
B | A B* C | 1 | 41 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | 45 | Yes
C | A B* C | 1 | 45 | Yes
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING [LEADER]
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable
All reported replicas are:
A = b6298d218143477eb0770540c0cdbbcc
B = f661a2fa62084188b9de0c0e1608bd8b
C = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 44 | Yes
B | A* B C | 1 | 44 | Yes
C | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 2 | 41 | Yes
C | A* B C | 2 | 41 | Yes
Tablet b800bde196ba429ea0b4a397a55cdaec of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): RUNNING [LEADER]
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): TS unavailable [NONVOTER]
All reported replicas are:
A = b6298d218143477eb0770540c0cdbbcc
B = f661a2fa62084188b9de0c0e1608bd8b
C = b3ac395387bb4601b607ae4a8404f325
D = c487cdc5c45c4365854a474ec10d6e54
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 45 | Yes
B | A* B C | 2 | 45 | Yes
C | A* B C | 2 | 45 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | CONSENSUS_MISMATCH | 20 | 1 | 0 | 17 | 2
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 3
Median | 12
Third Quartile | 19
Maximum | 19
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 64
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | HEALTHY | <none> | 0 | 0
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 12 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 1 | 0
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | HEALTHY | <none> | 0 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 0 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 8 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.22.231.1 | experimental | 127.22.231.1:42115
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.4 | experimental | 127.22.231.4:40461
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.231.1:42115
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.231.4:40461
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 13
Third Quartile | 18
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:12:21.300145 23452 raft_consensus_stress-itest.cc:258] completed iteration 0
F20240501 20:12:43.569144 26826 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.queue_time_us, tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.queue_time_us) (tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.run_cpu_time_us, tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.run_cpu_time_us) (tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.run_wall_time_us, tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.run_wall_time_us) (tablet-download-pool-0ea0b1495ddb472f9ee9ac06721a6482.queue_time_us, tablet-download-pool-0ea0b1495ddb472f9ee9ac06721a6482.queue_time_us) (tablet-download-pool-0ea0b1495ddb472f9ee9ac06721a6482.run_cpu_time_us, tablet-download-pool-0ea0b1495ddb472f9ee9ac06721a6482.run_cpu_time_us) (tablet-download-pool-0ea0b1495ddb472f9ee9ac06721a6482.run_wall_time_us, tablet-download-pool-0ea0b1495ddb472f9ee9ac06721a6482.run_wall_time_us) (tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.queue_time_us, tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.queue_time_us) (tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.run_cpu_time_us, tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.run_cpu_time_us) (tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.run_wall_time_us, tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.run_wall_time_us) (tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.queue_time_us, tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.queue_time_us) (tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.run_cpu_time_us, tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.run_cpu_time_us) (tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.run_wall_time_us, tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.run_wall_time_us) (tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.queue_time_us, tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.queue_time_us) (tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.run_cpu_time_us, tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.run_cpu_time_us) (tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.run_wall_time_us, tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.run_wall_time_us) (tablet-download-pool-35c85596a2564042a129626f13768a42.queue_time_us, tablet-download-pool-35c85596a2564042a129626f13768a42.queue_time_us) (tablet-download-pool-35c85596a2564042a129626f13768a42.run_cpu_time_us, tablet-download-pool-35c85596a2564042a129626f13768a42.run_cpu_time_us) (tablet-download-pool-35c85596a2564042a129626f13768a42.run_wall_time_us, tablet-download-pool-35c85596a2564042a129626f13768a42.run_wall_time_us) (tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.queue_time_us, tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.queue_time_us) (tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.run_cpu_time_us, tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.run_cpu_time_us) (tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.run_wall_time_us, tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.run_wall_time_us) (tablet-download-pool-6b64ee927588429eb0ace9f5a0241937.queue_time_us, tablet-download-pool-6b64ee927588429eb0ace9f5a0241937.queue_time_us) (tablet-download-pool-6b64ee927588429eb0ace9f5a0241937.run_cpu_time_us, tablet-download-pool-6b64ee927588429eb0ace9f5a0241937.run_cpu_time_us) (tablet-download-pool-6b64ee927588429eb0ace9f5a0241937.run_wall_time_us, tablet-download-pool-6b64ee927588429eb0ace9f5a0241937.run_wall_time_us) (tablet-download-pool-796121e2021b4939b7310e1216d34f21.queue_time_us, tablet-download-pool-796121e2021b4939b7310e1216d34f21.queue_time_us) (tablet-download-pool-796121e2021b4939b7310e1216d34f21.run_cpu_time_us, tablet-download-pool-796121e2021b4939b7310e1216d34f21.run_cpu_time_us) (tablet-download-pool-796121e2021b4939b7310e1216d34f21.run_wall_time_us, tablet-download-pool-796121e2021b4939b7310e1216d34f21.run_wall_time_us) (tablet-download-pool-8e89427782104f9288176ff6726bda56.queue_time_us, tablet-download-pool-8e89427782104f9288176ff6726bda56.queue_time_us) (tablet-download-pool-8e89427782104f9288176ff6726bda56.run_cpu_time_us, tablet-download-pool-8e89427782104f9288176ff6726bda56.run_cpu_time_us) (tablet-download-pool-8e89427782104f9288176ff6726bda56.run_wall_time_us, tablet-download-pool-8e89427782104f9288176ff6726bda56.run_wall_time_us) (tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.queue_time_us, tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.queue_time_us) (tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.run_cpu_time_us, tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.run_cpu_time_us) (tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.run_wall_time_us, tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.run_wall_time_us) (tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.queue_time_us, tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.queue_time_us) (tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.run_cpu_time_us, tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.run_cpu_time_us) (tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.run_wall_time_us, tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.run_wall_time_us) (tablet-download-pool-b800bde196ba429ea0b4a397a55cdaec.queue_time_us, tablet-download-pool-b800bde196ba429ea0b4a397a55cdaec.queue_time_us) (tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.queue_time_us, tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.queue_time_us) (tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.run_cpu_time_us, tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.run_cpu_time_us) (tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.run_wall_time_us, tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.run_wall_time_us) (tablet-download-pool-d5fda0ac4aa7417d91de600d14427af6.queue_time_us, tablet-download-pool-d5fda0ac4aa7417d91de600d14427af6.queue_time_us) (tablet-download-pool-d5fda0ac4aa7417d91de600d14427af6.run_cpu_time_us, tablet-download-pool-d5fda0ac4aa7417d91de600d14427af6.run_cpu_time_us) (tablet-download-pool-d5fda0ac4aa7417d91de600d14427af6.run_wall_time_us, tablet-download-pool-d5fda0ac4aa7417d91de600d14427af6.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f8901eabdcd google::LogMessage::Fail() at ??:0
@ 0x7f8901eafb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8901eab7cc google::LogMessage::Flush() at ??:0
@ 0x7f8901eacf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8904bec742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f8904bb8f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f8904bb2cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f89140058de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f891401ddaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f8914148480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f8914180a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f8914163981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f891416370e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f8904bbcb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8904bc019c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8904bc0019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f8904b9b01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8907f206db start_thread at ??:0
@ 0x7f8900d5e71f clone at ??:0
W20240501 20:12:45.566856 24368 meta_cache.cc:302] tablet 35c85596a2564042a129626f13768a42: replica 12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115) has failed: Network error: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111)
W20240501 20:12:51.881397 24368 meta_cache.cc:302] tablet 8e89427782104f9288176ff6726bda56: replica 12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115) has failed: Network error: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111)
F20240501 20:12:52.337822 27204 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.queue_time_us, tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.queue_time_us) (tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.run_cpu_time_us, tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.run_cpu_time_us) (tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.run_wall_time_us, tablet-download-pool-01b9a197f5464baf89ec25dd2c8d5867.run_wall_time_us) (tablet-download-pool-0ea0b1495ddb472f9ee9ac06721a6482.queue_time_us, tablet-download-pool-0ea0b1495ddb472f9ee9ac06721a6482.queue_time_us) (tablet-download-pool-1c7d48c373f349a98c71db9125837df7.queue_time_us, tablet-download-pool-1c7d48c373f349a98c71db9125837df7.queue_time_us) (tablet-download-pool-1c7d48c373f349a98c71db9125837df7.run_cpu_time_us, tablet-download-pool-1c7d48c373f349a98c71db9125837df7.run_cpu_time_us) (tablet-download-pool-1c7d48c373f349a98c71db9125837df7.run_wall_time_us, tablet-download-pool-1c7d48c373f349a98c71db9125837df7.run_wall_time_us) (tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.queue_time_us, tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.queue_time_us) (tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.run_cpu_time_us, tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.run_cpu_time_us) (tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.run_wall_time_us, tablet-download-pool-2b272055764c4b7e9eb9703d961e23c4.run_wall_time_us) (tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.queue_time_us, tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.queue_time_us) (tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.run_cpu_time_us, tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.run_cpu_time_us) (tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.run_wall_time_us, tablet-download-pool-314ae8aa44184d1da3aaabc0ab29093f.run_wall_time_us) (tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.queue_time_us, tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.queue_time_us) (tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.run_cpu_time_us, tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.run_cpu_time_us) (tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.run_wall_time_us, tablet-download-pool-31d1169f54f448cb90ce1d6467f5baf8.run_wall_time_us) (tablet-download-pool-35c85596a2564042a129626f13768a42.queue_time_us, tablet-download-pool-35c85596a2564042a129626f13768a42.queue_time_us) (tablet-download-pool-35c85596a2564042a129626f13768a42.run_cpu_time_us, tablet-download-pool-35c85596a2564042a129626f13768a42.run_cpu_time_us) (tablet-download-pool-35c85596a2564042a129626f13768a42.run_wall_time_us, tablet-download-pool-35c85596a2564042a129626f13768a42.run_wall_time_us) (tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.queue_time_us, tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.queue_time_us) (tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.run_cpu_time_us, tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.run_cpu_time_us) (tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.run_wall_time_us, tablet-download-pool-67861a86df1f4536984bc5b6c8eb5df2.run_wall_time_us) (tablet-download-pool-6d5cba1382424973929695143efa7e76.queue_time_us, tablet-download-pool-6d5cba1382424973929695143efa7e76.queue_time_us) (tablet-download-pool-6d5cba1382424973929695143efa7e76.run_cpu_time_us, tablet-download-pool-6d5cba1382424973929695143efa7e76.run_cpu_time_us) (tablet-download-pool-6d5cba1382424973929695143efa7e76.run_wall_time_us, tablet-download-pool-6d5cba1382424973929695143efa7e76.run_wall_time_us) (tablet-download-pool-796121e2021b4939b7310e1216d34f21.queue_time_us, tablet-download-pool-796121e2021b4939b7310e1216d34f21.queue_time_us) (tablet-download-pool-796121e2021b4939b7310e1216d34f21.run_cpu_time_us, tablet-download-pool-796121e2021b4939b7310e1216d34f21.run_cpu_time_us) (tablet-download-pool-796121e2021b4939b7310e1216d34f21.run_wall_time_us, tablet-download-pool-796121e2021b4939b7310e1216d34f21.run_wall_time_us) (tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.queue_time_us, tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.queue_time_us) (tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.run_cpu_time_us, tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.run_cpu_time_us) (tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.run_wall_time_us, tablet-download-pool-acf5a2b3f9ab411b8c8de017f8329c9f.run_wall_time_us) (tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.queue_time_us, tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.queue_time_us) (tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.run_cpu_time_us, tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.run_cpu_time_us) (tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.run_wall_time_us, tablet-download-pool-adf37d18ed2a46aba91c451eeae0e1f6.run_wall_time_us) (tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.queue_time_us, tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.queue_time_us) (tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.run_cpu_time_us, tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.run_cpu_time_us) (tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.run_wall_time_us, tablet-download-pool-c3fd8f7897174e74aaa841d017103cf2.run_wall_time_us) (tablet-download-pool-c48d3280db4642f78583237014733ae5.queue_time_us, tablet-download-pool-c48d3280db4642f78583237014733ae5.queue_time_us) (tablet-download-pool-c48d3280db4642f78583237014733ae5.run_cpu_time_us, tablet-download-pool-c48d3280db4642f78583237014733ae5.run_cpu_time_us) (tablet-download-pool-c48d3280db4642f78583237014733ae5.run_wall_time_us, tablet-download-pool-c48d3280db4642f78583237014733ae5.run_wall_time_us) (tablet-download-pool-e42879f1c5d04e33a082e6cbd154cabd.queue_time_us, tablet-download-pool-e42879f1c5d04e33a082e6cbd154cabd.queue_time_us) (tablet-download-pool-e42879f1c5d04e33a082e6cbd154cabd.run_cpu_time_us, tablet-download-pool-e42879f1c5d04e33a082e6cbd154cabd.run_cpu_time_us) (tablet-download-pool-e42879f1c5d04e33a082e6cbd154cabd.run_wall_time_us, tablet-download-pool-e42879f1c5d04e33a082e6cbd154cabd.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f23496ffdcd google::LogMessage::Fail() at ??:0
@ 0x7f2349703b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f23496ff7cc google::LogMessage::Flush() at ??:0
@ 0x7f2349700f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f234c440742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f234c40cf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f234c406cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f235b8598de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f235b871daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f235b99c480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f235b9d4a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f235b9b7981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f235b9b770e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f234c410b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f234c41419c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f234c414019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f234c3ef01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f234f7746db start_thread at ??:0
@ 0x7f23485b271f clone at ??:0
W20240501 20:12:52.687665 27235 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:12:52.577265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.231.4:40461 (local address 127.0.0.1:37036)
0501 20:12:52.578207 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0501 20:12:52.578228 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 20:12:52.578353 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:12:52.687299 (+108946us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.231.4:40461: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":794}
W20240501 20:12:55.782528 27251 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:12:52.599762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.231.5:33707 (local address 127.0.0.1:44650)
0501 20:12:52.599969 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0501 20:12:52.599986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 20:12:52.600068 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:12:55.782341 (+3182273us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.22.231.5:33707: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":111}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 1 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
Error from 127.22.231.5:33707: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.5:33707: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 1 other server(s)
Tablet Summary
Tablet 6b64ee927588429eb0ace9f5a0241937 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable [NONVOTER]
All reported replicas are:
A = 12bcc720148d426aa9ca25efe70ac6d4
B = c487cdc5c45c4365854a474ec10d6e54
C = aff8dec1c558403bb403fe2e08a34305
D = f661a2fa62084188b9de0c0e1608bd8b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 64 | Yes
C | A B* C D~ | 4 | 64 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [LEADER]
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 69 | Yes
B | [config not available] | | |
C | A B* C | 4 | 69 | Yes
Tablet 1c7d48c373f349a98c71db9125837df7 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [LEADER]
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 6 | 71 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 69 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 59 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 59 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 72 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 67 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 67 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 7 | 67 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 63 | Yes
B | A* B C | 2 | 63 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 66 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [LEADER]
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 8 | 74 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 65 | Yes
B | A* B C | 4 | 65 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 7 | 70 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 61 | Yes
B | A* B C | 2 | 61 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 72 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 61 | Yes
B | [config not available] | | |
C | A* B C | 7 | 61 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is unavailable: 2 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): TS unavailable [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 8 | 70 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [LEADER]
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 23 | Yes
B | [config not available] | | |
C | A B* C | 5 | 23 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNAVAILABLE | 20 | 3 | 0 | 8 | 9
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 16
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 66
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 3 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 20:12:58.198906 24367 meta_cache.cc:302] tablet 6d5cba1382424973929695143efa7e76: replica b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461) has failed: Network error: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 26 | Yes
B | [config not available] | | |
C | A B C* | 6 | 26 | Yes
Tablet 6b64ee927588429eb0ace9f5a0241937 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [NONVOTER]
All reported replicas are:
A = 12bcc720148d426aa9ca25efe70ac6d4
B = c487cdc5c45c4365854a474ec10d6e54
C = aff8dec1c558403bb403fe2e08a34305
D = f661a2fa62084188b9de0c0e1608bd8b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 4 | 66 | Yes
C | A B* C D~ | 4 | 66 | Yes
D | A B* C D~ | 4 | 66 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 73 | Yes
B | [config not available] | | |
C | A B C* | 5 | 73 | Yes
Tablet 1c7d48c373f349a98c71db9125837df7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 71 | Yes
B | [config not available] | | |
C | A* B C | 7 | 71 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 70 | Yes
B | A B* C D~ | 6 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 63 | Yes
B | [config not available] | | |
C | A B C* | 4 | 63 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 73 | Yes
B | A B* C D~ | 7 | 73 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 71 | Yes
B | [config not available] | | |
C | A B C* | 7 | 71 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 67 | Yes
B | A B* C | 8 | 67 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 67 | Yes
B | A* B C D~ | 2 | 67 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 68 | Yes
B | A B* C D~ | 4 | 68 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 74 | Yes
B | [config not available] | | |
C | A* B C | 9 | 74 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 69 | Yes
B | A* B C D~ | 4 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 70 | Yes
B | A B* C | 8 | 70 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 65 | Yes
B | A* B C D~ | 2 | 65 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 73 | Yes
B | A B* C D~ | 5 | 73 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 68 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 68 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 70 | Yes
B | A B* C | 9 | 70 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 3 | 0 | 17 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 72
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.5:33707, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 27 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 27 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 76 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 76 | Yes
D | [config not available] | | |
Tablet 1c7d48c373f349a98c71db9125837df7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 71 | Yes
B | [config not available] | | |
C | A* B C | 7 | 71 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 70 | Yes
B | A B* C D~ | 6 | 70 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 63 | Yes
B | [config not available] | | |
C | A B C* | 4 | 63 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 73 | Yes
B | A B* C D~ | 7 | 73 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 71 | Yes
B | [config not available] | | |
C | A B C* | 7 | 71 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 67 | Yes
B | A B* C | 8 | 67 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 67 | Yes
B | A* B C D~ | 2 | 67 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 68 | Yes
B | A B* C D~ | 4 | 68 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 74 | Yes
B | [config not available] | | |
C | A* B C | 9 | 74 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 69 | Yes
B | A* B C D~ | 4 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 70 | Yes
B | A B* C | 8 | 70 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 65 | Yes
B | A* B C D~ | 2 | 65 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 73 | Yes
B | A B* C D~ | 5 | 73 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 68 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 68 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 70 | Yes
B | A B* C | 9 | 70 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 4 | 0 | 16 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 76 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 76 | Yes
D | [config not available] | | |
Tablet 1c7d48c373f349a98c71db9125837df7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet 1c7d48c373f349a98c71db9125837df7: Tablet copy aborted
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 79 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 79 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 77 | Yes
B | A B* C | 6 | 77 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 63 | Yes
B | [config not available] | | |
C | A B C* | 4 | 63 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 80 | Yes
B | A B* C | 7 | 80 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 71 | Yes
B | [config not available] | | |
C | A B C* | 7 | 71 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 67 | Yes
B | A B* C | 8 | 67 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 67 | Yes
B | A* B C D~ | 2 | 67 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 75 | Yes
B | A B* C | 4 | 75 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 81 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 81 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 69 | Yes
B | A* B C D~ | 4 | 69 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 70 | Yes
B | A B* C | 8 | 70 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 65 | Yes
B | A* B C D~ | 2 | 65 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 80 | Yes
B | A B* C | 5 | 80 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 68 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 68 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 70 | Yes
B | A B* C | 9 | 70 | Yes
C | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 27 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 27 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 4 | 0 | 16 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 6 | 28 | Yes
B | [config not available] | | |
C | A B C* | 6 | 28 | No
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 76 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 76 | Yes
D | [config not available] | | |
Tablet 1c7d48c373f349a98c71db9125837df7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 79 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 79 | Yes
D | A* B C D~ | 7 | 79 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 77 | Yes
B | A B* C | 6 | 77 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 69 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 69 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 80 | Yes
B | A B* C | 7 | 80 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 76 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 76 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 73 | Yes
B | A B* C D~ | 8 | 73 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 73 | Yes
B | A* B C | 2 | 73 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 75 | Yes
B | A B* C | 4 | 75 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 81 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 81 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 75 | Yes
B | A* B C | 4 | 75 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 77 | Yes
B | A B* C D~ | 8 | 77 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 70 | Yes
B | A* B C | 2 | 70 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 80 | Yes
B | A B* C | 5 | 80 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 73 | Yes
B | [config not available] | | |
C | A* B C | 7 | 73 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 79 | Yes
B | A B* C D~ | 9 | 79 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 4 | 0 | 16 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 16
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 72
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 28 | Yes
B | [config not available] | | |
C | A B C* | 6 | 28 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 82 | Yes
B | [config not available] | | |
C | A B C* | 5 | 82 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 77 | Yes
B | A B* C | 6 | 77 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 69 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 69 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 80 | Yes
B | A B* C | 7 | 80 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 76 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 76 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 73 | Yes
B | A B* C D~ | 8 | 73 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 73 | Yes
B | A* B C | 2 | 73 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 75 | Yes
B | A B* C | 4 | 75 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 81 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 81 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 75 | Yes
B | A* B C | 4 | 75 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 77 | Yes
B | A B* C D~ | 8 | 77 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 70 | Yes
B | A* B C | 2 | 70 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 80 | Yes
B | A B* C | 5 | 80 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 73 | Yes
B | [config not available] | | |
C | A* B C | 7 | 73 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 79 | Yes
B | A B* C D~ | 9 | 79 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 7
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 69
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.5:33707, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 28 | Yes
B | [config not available] | | |
C | A B C* | 6 | 28 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 82 | Yes
B | [config not available] | | |
C | A B C* | 5 | 82 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 82 | Yes
B | A B* C D~ | 6 | 82 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 4 | 76 | Yes
B | [config not available] | | |
C | A B C* | 4 | 76 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 85 | Yes
B | A B* C D~ | 7 | 85 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 7 | 82 | Yes
B | [config not available] | | |
C | A B C* | 7 | 82 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 79 | Yes
B | A B* C | 8 | 79 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 73 | Yes
B | A* B C | 2 | 73 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 80 | Yes
B | A B* C D~ | 4 | 80 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 85 | Yes
B | [config not available] | | |
C | A* B C | 9 | 85 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 81 | Yes
B | A* B C D~ | 4 | 81 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 84 | Yes
B | A B* C | 8 | 84 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 77 | Yes
B | A* B C D~ | 2 | 77 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 84 | Yes
B | A B* C D~ | 5 | 84 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 73 | Yes
B | [config not available] | | |
C | A* B C | 7 | 73 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 9 | 85 | Yes
B | A B* C | 9 | 85 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 72
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 29 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 29 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 89 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 89 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 82 | Yes
B | A B* C D~ | 6 | 82 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 76 | Yes
B | [config not available] | | |
C | A B C* | 4 | 76 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 85 | Yes
B | A B* C D~ | 7 | 85 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 82 | Yes
B | [config not available] | | |
C | A B C* | 7 | 82 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 79 | Yes
B | A B* C | 8 | 79 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 79 | Yes
B | A* B C D~ | 2 | 79 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 80 | Yes
B | A B* C D~ | 4 | 80 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 85 | Yes
B | [config not available] | | |
C | A* B C | 9 | 85 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 81 | Yes
B | A* B C D~ | 4 | 81 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 84 | Yes
B | A B* C | 8 | 84 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 77 | Yes
B | A* B C D~ | 2 | 77 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 84 | Yes
B | A B* C D~ | 5 | 84 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 78 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 78 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 85 | Yes
B | A B* C | 9 | 85 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 29 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 29 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 89 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 89 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 82 | Yes
B | A B* C D~ | 6 | 82 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 76 | Yes
B | [config not available] | | |
C | A B C* | 4 | 76 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 85 | Yes
B | A B* C D~ | 7 | 85 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 82 | Yes
B | [config not available] | | |
C | A B C* | 7 | 82 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 79 | Yes
B | A B* C | 8 | 79 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 79 | Yes
B | A* B C D~ | 2 | 79 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 80 | Yes
B | A B* C D~ | 4 | 80 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 85 | Yes
B | [config not available] | | |
C | A* B C | 9 | 85 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 81 | Yes
B | A* B C D~ | 4 | 81 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 84 | Yes
B | A B* C | 8 | 84 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 77 | Yes
B | A* B C D~ | 2 | 77 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 84 | Yes
B | A B* C D~ | 5 | 84 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 78 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 78 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 85 | Yes
B | A B* C | 9 | 85 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 29 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 29 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 89 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 89 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 89 | Yes
B | A B* C | 6 | 89 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 76 | Yes
B | [config not available] | | |
C | A B C* | 4 | 76 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 92 | Yes
B | A B* C | 7 | 92 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 82 | Yes
B | [config not available] | | |
C | A B C* | 7 | 82 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 79 | Yes
B | A B* C | 8 | 79 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 79 | Yes
B | A* B C D~ | 2 | 79 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 88 | Yes
B | A B* C | 4 | 88 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 92 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 92 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 81 | Yes
B | A* B C D~ | 4 | 81 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 8 | 90 | Yes
B | A B* C D~ | 8 | 90 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 83 | Yes
B | A* B C | 2 | 83 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 90 | Yes
B | A B* C | 5 | 90 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 78 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 78 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 85 | Yes
B | A B* C | 9 | 85 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 70
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 29 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 29 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 89 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 89 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 89 | Yes
B | A B* C | 6 | 89 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 83 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 83 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 92 | Yes
B | A B* C | 7 | 92 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 89 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 89 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 86 | Yes
B | A B* C D~ | 8 | 86 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 86 | Yes
B | A* B C | 2 | 86 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 88 | Yes
B | A B* C | 4 | 88 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 92 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 92 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 88 | Yes
B | A* B C | 4 | 88 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 90 | Yes
B | A B* C D~ | 8 | 90 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 83 | Yes
B | A* B C | 2 | 83 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 90 | Yes
B | A B* C | 5 | 90 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 84 | Yes
B | [config not available] | | |
C | A* B C | 7 | 84 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 92 | Yes
B | A B* C D~ | 9 | 92 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 30 | Yes
B | [config not available] | | |
C | A B C* | 6 | 30 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 95 | Yes
B | [config not available] | | |
C | A B C* | 5 | 95 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 89 | Yes
B | A B* C | 6 | 89 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 83 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 83 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 92 | Yes
B | A B* C | 7 | 92 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 89 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 89 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 86 | Yes
B | A B* C D~ | 8 | 86 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 86 | Yes
B | A* B C | 2 | 86 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 88 | Yes
B | A B* C | 4 | 88 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 92 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 92 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 88 | Yes
B | A* B C | 4 | 88 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 90 | Yes
B | A B* C D~ | 8 | 90 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 83 | Yes
B | A* B C | 2 | 83 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 90 | Yes
B | A B* C | 5 | 90 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 84 | Yes
B | [config not available] | | |
C | A* B C | 7 | 84 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 92 | Yes
B | A B* C D~ | 9 | 92 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 7
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 69
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.2:39099, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 30 | Yes
B | [config not available] | | |
C | A B C* | 6 | 30 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 95 | Yes
B | [config not available] | | |
C | A B C* | 5 | 95 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 89 | Yes
B | A B* C | 6 | 89 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 83 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 83 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 92 | Yes
B | A B* C | 7 | 92 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 89 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 89 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 86 | Yes
B | A B* C D~ | 8 | 86 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 86 | Yes
B | A* B C | 2 | 86 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 88 | Yes
B | A B* C | 4 | 88 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 99 | Yes
B | [config not available] | | |
C | A* B C | 9 | 99 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 88 | Yes
B | A* B C | 4 | 88 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 90 | Yes
B | A B* C D~ | 8 | 90 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 83 | Yes
B | A* B C | 2 | 83 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 90 | Yes
B | A B* C | 5 | 90 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 84 | Yes
B | [config not available] | | |
C | A* B C | 7 | 84 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 92 | Yes
B | A B* C D~ | 9 | 92 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 68
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.2:39099, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 30 | Yes
B | [config not available] | | |
C | A B C* | 6 | 30 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 95 | Yes
B | [config not available] | | |
C | A B C* | 5 | 95 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 97 | Yes
B | A B* C D~ | 6 | 97 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 90 | Yes
B | [config not available] | | |
C | A B C* | 4 | 90 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 100 | Yes
B | A B* C D~ | 7 | 100 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 97 | Yes
B | [config not available] | | |
C | A B C* | 7 | 97 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 93 | Yes
B | A B* C | 8 | 93 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 94 | Yes
B | A* B C D~ | 2 | 94 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 96 | Yes
B | A B* C D~ | 4 | 96 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 99 | Yes
B | [config not available] | | |
C | A* B C | 9 | 99 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 97 | Yes
B | A* B C D~ | 4 | 97 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 96 | Yes
B | A B* C | 8 | 96 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 91 | Yes
B | A* B C D~ | 2 | 91 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 98 | Yes
B | A B* C D~ | 5 | 98 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 93 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 93 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 101 | Yes
B | A B* C | 9 | 101 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 103 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 103 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 97 | Yes
B | A B* C D~ | 6 | 97 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 90 | Yes
B | [config not available] | | |
C | A B C* | 4 | 90 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 100 | Yes
B | A B* C D~ | 7 | 100 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 97 | Yes
B | [config not available] | | |
C | A B C* | 7 | 97 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 93 | Yes
B | A B* C | 8 | 93 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 94 | Yes
B | A* B C D~ | 2 | 94 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 96 | Yes
B | A B* C D~ | 4 | 96 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 99 | Yes
B | [config not available] | | |
C | A* B C | 9 | 99 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 97 | Yes
B | A* B C D~ | 4 | 97 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 96 | Yes
B | A B* C | 8 | 96 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 91 | Yes
B | A* B C D~ | 2 | 91 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 98 | Yes
B | A B* C D~ | 5 | 98 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 93 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 93 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 101 | Yes
B | A B* C | 9 | 101 | Yes
C | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 31 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 31 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 31 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 31 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 103 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 103 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 6 | 103 | Yes
B | A B* C | 6 | 103 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 90 | Yes
B | [config not available] | | |
C | A B C* | 4 | 90 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 7 | 107 | Yes
B | A B* C | 7 | 107 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 97 | Yes
B | [config not available] | | |
C | A B C* | 7 | 97 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 93 | Yes
B | A B* C | 8 | 93 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 94 | Yes
B | A* B C D~ | 2 | 94 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 96 | Yes
B | A B* C D~ | 4 | 96 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 105 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 105 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 97 | Yes
B | A* B C D~ | 4 | 97 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 96 | Yes
B | A B* C | 8 | 96 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 91 | Yes
B | A* B C D~ | 2 | 91 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 104 | Yes
B | A B* C | 5 | 104 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 93 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 93 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 101 | Yes
B | A B* C | 9 | 101 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.5:33707, tserver@127.22.231.2:39099, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 31 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 31 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 103 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 103 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 103 | Yes
B | A B* C | 6 | 103 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 94 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 94 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 107 | Yes
B | A B* C | 7 | 107 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 103 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 103 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 99 | Yes
B | A B* C D~ | 8 | 99 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 98 | Yes
B | A* B C | 2 | 98 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 103 | Yes
B | A B* C | 4 | 103 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 105 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 105 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 102 | Yes
B | A* B C | 4 | 102 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 103 | Yes
B | A B* C D~ | 8 | 103 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 97 | Yes
B | A* B C | 2 | 97 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 104 | Yes
B | A B* C | 5 | 104 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 99 | Yes
B | [config not available] | | |
C | A* B C | 7 | 99 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 107 | Yes
B | A B* C D~ | 9 | 107 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 109 | Yes
B | [config not available] | | |
C | A B C* | 5 | 109 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 103 | Yes
B | A B* C | 6 | 103 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 94 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 94 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 107 | Yes
B | A B* C | 7 | 107 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 103 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 103 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 99 | Yes
B | A B* C D~ | 8 | 99 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 98 | Yes
B | A* B C | 2 | 98 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 103 | Yes
B | A B* C | 4 | 103 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 105 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 105 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 102 | Yes
B | A* B C | 4 | 102 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 103 | Yes
B | A B* C D~ | 8 | 103 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 97 | Yes
B | A* B C | 2 | 97 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 104 | Yes
B | A B* C | 5 | 104 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 99 | Yes
B | [config not available] | | |
C | A* B C | 7 | 99 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 107 | Yes
B | A B* C D~ | 9 | 107 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 32 | Yes
B | [config not available] | | |
C | A B C* | 6 | 32 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 7
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 69
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 32 | Yes
B | [config not available] | | |
C | A B C* | 6 | 32 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 109 | Yes
B | [config not available] | | |
C | A B C* | 5 | 109 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 110 | Yes
B | A B* C D~ | 6 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 94 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 94 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 111 | Yes
B | A B* C D~ | 7 | 111 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 103 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 103 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 99 | Yes
B | A B* C | 8 | 105 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 98 | Yes
B | A* B C | 2 | 98 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 103 | Yes
B | A B* C | 4 | 103 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 111 | Yes
B | [config not available] | | |
C | A* B C | 9 | 111 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 102 | Yes
B | A* B C | 4 | 102 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 110 | Yes
B | A B* C | 8 | 110 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 97 | Yes
B | A* B C | 2 | 97 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 111 | Yes
B | A B* C D~ | 5 | 111 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 99 | Yes
B | [config not available] | | |
C | A* B C | 7 | 99 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 107 | Yes
B | A B* C D~ | 9 | 107 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 8
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 70
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.2:39099, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 32 | Yes
B | [config not available] | | |
C | A B C* | 6 | 32 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 109 | Yes
B | [config not available] | | |
C | A B C* | 5 | 109 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 110 | Yes
B | A B* C D~ | 6 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 101 | Yes
B | [config not available] | | |
C | A B C* | 4 | 101 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 111 | Yes
B | A B* C D~ | 7 | 111 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 110 | Yes
B | [config not available] | | |
C | A B C* | 7 | 110 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 105 | Yes
B | A B* C | 8 | 105 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 104 | Yes
B | A* B C D~ | 2 | 104 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 110 | Yes
B | A B* C D~ | 4 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 111 | Yes
B | [config not available] | | |
C | A* B C | 9 | 111 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 109 | Yes
B | A* B C D~ | 4 | 109 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 110 | Yes
B | A B* C | 8 | 110 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 104 | Yes
B | A* B C D~ | 2 | 104 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 111 | Yes
B | A B* C D~ | 5 | 111 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 106 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 106 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 114 | Yes
B | A B* C | 9 | 114 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 33 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 33 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 116 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 116 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 110 | Yes
B | A B* C D~ | 6 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 101 | Yes
B | [config not available] | | |
C | A B C* | 4 | 101 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 111 | Yes
B | A B* C D~ | 7 | 111 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 110 | Yes
B | [config not available] | | |
C | A B C* | 7 | 110 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 105 | Yes
B | A B* C | 8 | 105 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 104 | Yes
B | A* B C D~ | 2 | 104 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 110 | Yes
B | A B* C D~ | 4 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 111 | Yes
B | [config not available] | | |
C | A* B C | 9 | 111 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 109 | Yes
B | A* B C D~ | 4 | 109 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 110 | Yes
B | A B* C | 8 | 110 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 104 | Yes
B | A* B C D~ | 2 | 104 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 111 | Yes
B | A B* C D~ | 5 | 111 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 106 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 106 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 114 | Yes
B | A B* C | 9 | 114 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 116 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 116 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 117 | Yes
B | A B* C | 6 | 117 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 101 | Yes
B | [config not available] | | |
C | A B C* | 4 | 101 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 118 | Yes
B | A B* C | 7 | 118 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 110 | Yes
B | [config not available] | | |
C | A B C* | 7 | 110 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 105 | Yes
B | A B* C | 8 | 105 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 104 | Yes
B | A* B C D~ | 2 | 104 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 110 | Yes
B | A B* C D~ | 4 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 118 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 118 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 109 | Yes
B | A* B C D~ | 4 | 109 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 117 | Yes
B | A B* C D~ | 8 | 117 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 104 | Yes
B | A* B C D~ | 2 | 104 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 117 | Yes
B | A B* C | 5 | 117 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 106 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 106 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 114 | Yes
B | A B* C | 9 | 114 | Yes
C | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 33 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 33 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 72
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 33 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 33 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 116 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 116 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 117 | Yes
B | A B* C | 6 | 117 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 107 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 107 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 118 | Yes
B | A B* C | 7 | 118 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 118 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 118 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 110 | Yes
B | A B* C D~ | 8 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 111 | Yes
B | A* B C | 2 | 111 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 116 | Yes
B | A B* C | 4 | 116 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 118 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 118 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 116 | Yes
B | A* B C | 4 | 116 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 117 | Yes
B | A B* C D~ | 8 | 117 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 109 | Yes
B | A* B C | 2 | 109 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 117 | Yes
B | A B* C | 5 | 117 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 113 | Yes
B | [config not available] | | |
C | A* B C | 7 | 113 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 120 | Yes
B | A B* C D~ | 9 | 120 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 34 | Yes
B | [config not available] | | |
C | A B C* | 6 | 34 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 124 | Yes
B | [config not available] | | |
C | A B C* | 5 | 124 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 117 | Yes
B | A B* C | 6 | 117 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 107 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 107 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 118 | Yes
B | A B* C | 7 | 118 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 118 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 118 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 110 | Yes
B | A B* C D~ | 8 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 111 | Yes
B | A* B C | 2 | 111 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 116 | Yes
B | A B* C | 4 | 116 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 118 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 118 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 116 | Yes
B | A* B C | 4 | 116 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 117 | Yes
B | A B* C D~ | 8 | 117 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 109 | Yes
B | A* B C | 2 | 109 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 117 | Yes
B | A B* C | 5 | 117 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 113 | Yes
B | [config not available] | | |
C | A* B C | 7 | 113 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 120 | Yes
B | A B* C D~ | 9 | 120 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 7
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 69
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 124 | Yes
B | [config not available] | | |
C | A B C* | 5 | 124 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 124 | Yes
B | A B* C D~ | 6 | 124 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 107 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 107 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 125 | Yes
B | A B* C D~ | 7 | 125 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 118 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 118 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 110 | Yes
B | A B* C D~ | 8 | 110 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 111 | Yes
B | A* B C | 2 | 111 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 116 | Yes
B | A B* C | 4 | 116 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 125 | Yes
B | [config not available] | | |
C | A* B C | 9 | 125 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 116 | Yes
B | A* B C | 4 | 116 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 123 | Yes
B | A B* C | 8 | 123 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 2 | 115 | Yes
B | A* B C D~ | 2 | 115 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 124 | Yes
B | A B* C D~ | 5 | 124 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 113 | Yes
B | [config not available] | | |
C | A* B C | 7 | 113 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 120 | Yes
B | A B* C D~ | 9 | 120 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 34 | Yes
B | [config not available] | | |
C | A B C* | 6 | 34 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 8
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 70
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 124 | Yes
B | [config not available] | | |
C | A B C* | 5 | 124 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 124 | Yes
B | A B* C D~ | 6 | 124 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 112 | Yes
B | [config not available] | | |
C | A B C* | 4 | 112 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 125 | Yes
B | A B* C D~ | 7 | 125 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 125 | Yes
B | [config not available] | | |
C | A B C* | 7 | 125 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 116 | Yes
B | A B* C | 8 | 116 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 118 | Yes
B | A* B C D~ | 2 | 118 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 123 | Yes
B | A B* C D~ | 4 | 123 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 125 | Yes
B | [config not available] | | |
C | A* B C | 9 | 125 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 123 | Yes
B | A* B C D~ | 4 | 123 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 123 | Yes
B | A B* C | 8 | 123 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 115 | Yes
B | A* B C D~ | 2 | 115 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 124 | Yes
B | A B* C D~ | 5 | 124 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 120 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 120 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 126 | Yes
B | A B* C | 9 | 126 | Yes
C | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 34 | Yes
B | [config not available] | | |
C | A B C* | 6 | 34 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 35 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 35 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 131 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 131 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 124 | Yes
B | A B* C D~ | 6 | 124 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 112 | Yes
B | [config not available] | | |
C | A B C* | 4 | 112 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 125 | Yes
B | A B* C D~ | 7 | 125 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 125 | Yes
B | [config not available] | | |
C | A B C* | 7 | 125 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 116 | Yes
B | A B* C | 8 | 116 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 118 | Yes
B | A* B C D~ | 2 | 118 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 123 | Yes
B | A B* C D~ | 4 | 123 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 9 | 132 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 132 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 123 | Yes
B | A* B C D~ | 4 | 123 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 123 | Yes
B | A B* C | 8 | 123 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 115 | Yes
B | A* B C D~ | 2 | 115 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 124 | Yes
B | A B* C D~ | 5 | 124 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 120 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 120 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 126 | Yes
B | A B* C | 9 | 126 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.2:39099, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 131 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 131 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 131 | Yes
B | A B* C | 6 | 131 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 119 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 119 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 131 | Yes
B | A B* C | 7 | 131 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 125 | Yes
B | [config not available] | | |
C | A B C* | 7 | 125 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 116 | Yes
B | A B* C | 8 | 116 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 118 | Yes
B | A* B C D~ | 2 | 118 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 130 | Yes
B | A B* C | 4 | 130 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 132 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 132 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 4 | 130 | Yes
B | A* B C | 4 | 130 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 130 | Yes
B | A B* C D~ | 8 | 130 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 122 | Yes
B | A* B C | 2 | 122 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 129 | Yes
B | A B* C | 5 | 129 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 120 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 120 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 126 | Yes
B | A B* C | 9 | 126 | Yes
C | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 35 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 35 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.2:39099, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 35 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 35 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 131 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 131 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 131 | Yes
B | A B* C | 6 | 131 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 119 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 119 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 131 | Yes
B | A B* C | 7 | 131 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 131 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 131 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 123 | Yes
B | A B* C D~ | 8 | 123 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 124 | Yes
B | A* B C | 2 | 124 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 130 | Yes
B | A B* C | 4 | 130 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 132 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 132 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 130 | Yes
B | A* B C | 4 | 130 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 130 | Yes
B | A B* C D~ | 8 | 130 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 122 | Yes
B | A* B C | 2 | 122 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 129 | Yes
B | A B* C | 5 | 129 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 125 | Yes
B | [config not available] | | |
C | A* B C | 7 | 125 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 133 | Yes
B | A B* C D~ | 9 | 133 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.2:39099, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 36 | Yes
B | [config not available] | | |
C | A B C* | 6 | 36 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 137 | Yes
B | [config not available] | | |
C | A B C* | 5 | 137 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 131 | Yes
B | A B* C | 6 | 131 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 119 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 119 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 131 | Yes
B | A B* C | 7 | 131 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 131 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 131 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 123 | Yes
B | A B* C D~ | 8 | 123 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 124 | Yes
B | A* B C | 2 | 124 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 130 | Yes
B | A B* C | 4 | 130 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 137 | Yes
B | [config not available] | | |
C | A* B C | 9 | 137 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 130 | Yes
B | A* B C | 4 | 130 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 130 | Yes
B | A B* C D~ | 8 | 130 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 122 | Yes
B | A* B C | 2 | 122 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 129 | Yes
B | A B* C | 5 | 129 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 125 | Yes
B | [config not available] | | |
C | A* B C | 7 | 125 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 133 | Yes
B | A B* C D~ | 9 | 133 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 68
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.2:39099, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 36 | Yes
B | [config not available] | | |
C | A B C* | 6 | 36 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 137 | Yes
B | [config not available] | | |
C | A B C* | 5 | 137 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 137 | Yes
B | A B* C D~ | 6 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 124 | Yes
B | [config not available] | | |
C | A B C* | 4 | 124 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 137 | Yes
B | A B* C D~ | 7 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 131 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 131 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 8 | 129 | Yes
B | A B* C | 8 | 129 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 2 | 131 | Yes
B | A* B C D~ | 2 | 131 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 137 | Yes
B | A B* C D~ | 4 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 137 | Yes
B | [config not available] | | |
C | A* B C | 9 | 137 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 136 | Yes
B | A* B C D~ | 4 | 136 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 136 | Yes
B | A B* C | 8 | 136 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 128 | Yes
B | A* B C D~ | 2 | 128 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 135 | Yes
B | A B* C D~ | 5 | 135 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 125 | Yes
B | [config not available] | | |
C | A* B C | 7 | 125 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 9 | 140 | Yes
B | A B* C | 9 | 140 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 72
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* D~ | 6 | 37 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 37 | No
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* D~ | 5 | 144 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 144 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 137 | Yes
B | A B* C D~ | 6 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 124 | Yes
B | [config not available] | | |
C | A B C* | 4 | 124 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 137 | Yes
B | A B* C D~ | 7 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 137 | Yes
B | [config not available] | | |
C | A B C* | 7 | 137 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 129 | Yes
B | A B* C | 8 | 129 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 131 | Yes
B | A* B C D~ | 2 | 131 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 137 | Yes
B | A B* C D~ | 4 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 137 | Yes
B | [config not available] | | |
C | A* B C | 9 | 137 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 136 | Yes
B | A* B C D~ | 4 | 136 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 136 | Yes
B | A B* C | 8 | 136 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 128 | Yes
B | A* B C D~ | 2 | 128 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 135 | Yes
B | A B* C D~ | 5 | 135 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 130 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 130 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 140 | Yes
B | A B* C | 9 | 140 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 37 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 37 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 144 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 144 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 137 | Yes
B | A B* C D~ | 6 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 124 | Yes
B | [config not available] | | |
C | A B C* | 4 | 124 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 7 | 143 | Yes
B | A B* C | 7 | 143 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 137 | Yes
B | [config not available] | | |
C | A B C* | 7 | 137 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 129 | Yes
B | A B* C | 8 | 129 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 131 | Yes
B | A* B C D~ | 2 | 131 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 137 | Yes
B | A B* C D~ | 4 | 137 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 141 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 141 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 136 | Yes
B | A* B C D~ | 4 | 136 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 140 | Yes
B | A B* C D~ | 8 | 140 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 128 | Yes
B | A* B C D~ | 2 | 128 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 141 | Yes
B | A B* C | 5 | 141 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 130 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 130 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 140 | Yes
B | A B* C | 9 | 140 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 13
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 74
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 37 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 37 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 144 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 144 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 143 | Yes
B | A B* C | 6 | 143 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 130 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 130 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 143 | Yes
B | A B* C | 7 | 143 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* D~ | 7 | 144 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 144 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 135 | Yes
B | A B* C D~ | 8 | 135 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 137 | Yes
B | A* B C | 2 | 137 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 142 | Yes
B | A B* C | 4 | 142 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 141 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 141 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 141 | Yes
B | A* B C | 4 | 141 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 140 | Yes
B | A B* C D~ | 8 | 140 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 134 | Yes
B | A* B C | 2 | 134 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 141 | Yes
B | A B* C | 5 | 141 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 137 | Yes
B | [config not available] | | |
C | A* B C | 7 | 137 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 146 | Yes
B | A B* C D~ | 9 | 146 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 70
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 38 | Yes
B | [config not available] | | |
C | A B C* | 6 | 38 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 148 | Yes
B | [config not available] | | |
C | A B C* | 5 | 148 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 143 | Yes
B | A B* C | 6 | 143 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 130 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 130 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 143 | Yes
B | A B* C | 7 | 143 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 144 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 144 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 135 | Yes
B | A B* C D~ | 8 | 135 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 137 | Yes
B | A* B C | 2 | 137 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 142 | Yes
B | A B* C | 4 | 142 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 148 | Yes
B | [config not available] | | |
C | A* B C | 9 | 148 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 141 | Yes
B | A* B C | 4 | 141 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 140 | Yes
B | A B* C D~ | 8 | 140 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 134 | Yes
B | A* B C | 2 | 134 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 141 | Yes
B | A B* C | 5 | 141 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 137 | Yes
B | [config not available] | | |
C | A* B C | 7 | 137 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 146 | Yes
B | A B* C D~ | 9 | 146 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 68
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 38 | Yes
B | [config not available] | | |
C | A B C* | 6 | 38 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 148 | Yes
B | [config not available] | | |
C | A B C* | 5 | 148 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 150 | Yes
B | A B* C D~ | 6 | 150 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 137 | Yes
B | [config not available] | | |
C | A B C* | 4 | 137 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 147 | Yes
B | A B* C D~ | 7 | 147 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 144 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 144 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 135 | Yes
B | A B* C D~ | 8 | 135 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 137 | Yes
B | A* B C | 2 | 137 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 148 | Yes
B | A B* C D~ | 4 | 148 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 148 | Yes
B | [config not available] | | |
C | A* B C | 9 | 148 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 148 | Yes
B | A* B C D~ | 4 | 148 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 144 | Yes
B | A B* C | 8 | 144 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 139 | Yes
B | A* B C D~ | 2 | 139 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 145 | Yes
B | A B* C D~ | 5 | 145 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 137 | Yes
B | [config not available] | | |
C | A* B C | 7 | 137 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 146 | Yes
B | A B* C D~ | 9 | 146 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 72
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 38 | Yes
B | [config not available] | | |
C | A B C* | 6 | 38 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 148 | Yes
B | [config not available] | | |
C | A B C* | 5 | 148 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 150 | Yes
B | A B* C D~ | 6 | 150 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 137 | Yes
B | [config not available] | | |
C | A B C* | 4 | 137 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 147 | Yes
B | A B* C D~ | 7 | 147 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 149 | Yes
B | [config not available] | | |
C | A B C* | 7 | 149 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 141 | Yes
B | A B* C | 8 | 141 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 143 | Yes
B | A* B C D~ | 2 | 143 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 148 | Yes
B | A B* C D~ | 4 | 148 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 148 | Yes
B | [config not available] | | |
C | A* B C | 9 | 148 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 148 | Yes
B | A* B C D~ | 4 | 148 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 144 | Yes
B | A B* C | 8 | 144 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 139 | Yes
B | A* B C D~ | 2 | 139 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 145 | Yes
B | A B* C D~ | 5 | 145 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 142 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 142 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 152 | Yes
B | A B* C | 9 | 152 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 153 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 153 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 150 | Yes
B | A B* C D~ | 6 | 150 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 137 | Yes
B | [config not available] | | |
C | A B C* | 4 | 137 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 147 | Yes
B | A B* C | 7 | 151 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 149 | Yes
B | [config not available] | | |
C | A B C* | 7 | 149 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 141 | Yes
B | A B* C | 8 | 141 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 143 | Yes
B | A* B C D~ | 2 | 143 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 148 | Yes
B | A B* C D~ | 4 | 148 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 153 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 153 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 148 | Yes
B | A* B C D~ | 4 | 148 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 8 | 151 | Yes
B | A B* C D~ | 8 | 151 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 139 | Yes
B | A* B C D~ | 2 | 139 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 152 | Yes
B | A B* C | 5 | 152 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 142 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 142 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 152 | Yes
B | A B* C | 9 | 152 | Yes
C | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 39 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 39 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 39 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 39 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 153 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 153 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 155 | Yes
B | A B* C | 6 | 155 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 142 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 142 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 151 | Yes
B | A B* C | 7 | 151 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 149 | Yes
B | [config not available] | | |
C | A B C* | 7 | 149 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 8 | 147 | Yes
B | A B* C D~ | 8 | 147 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 2 | 150 | Yes
B | A* B C | 2 | 150 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 153 | Yes
B | A B* C | 4 | 153 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 153 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 153 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 153 | Yes
B | A* B C | 4 | 153 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 151 | Yes
B | A B* C D~ | 8 | 151 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 144 | Yes
B | A* B C | 2 | 144 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 152 | Yes
B | A B* C | 5 | 152 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 7 | 149 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 142 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C D~ | 9 | 159 | Yes
B | A B* C D~ | 9 | 159 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 8
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 70
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 40 | Yes
B | [config not available] | | |
C | A B C* | 6 | 40 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* | 5 | 160 | Yes
B | [config not available] | | |
C | A B C* | 5 | 160 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 155 | Yes
B | A B* C | 6 | 155 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 142 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 142 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 151 | Yes
B | A B* C | 7 | 151 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 155 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 155 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 147 | Yes
B | A B* C D~ | 8 | 147 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 150 | Yes
B | A* B C | 2 | 150 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 153 | Yes
B | A B* C | 4 | 153 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 153 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 153 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 153 | Yes
B | A* B C | 4 | 153 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 151 | Yes
B | A B* C D~ | 8 | 151 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 144 | Yes
B | A* B C | 2 | 144 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 152 | Yes
B | A B* C | 5 | 152 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 149 | Yes
B | [config not available] | | |
C | A* B C | 7 | 149 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 159 | Yes
B | A B* C D~ | 9 | 159 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 8
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 70
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 40 | Yes
B | [config not available] | | |
C | A B C* | 6 | 40 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 160 | Yes
B | [config not available] | | |
C | A B C* | 5 | 160 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 155 | Yes
B | A B* C | 6 | 155 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 142 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 142 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 151 | Yes
B | A B* C | 7 | 151 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 155 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 155 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 147 | Yes
B | A B* C D~ | 8 | 147 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 150 | Yes
B | A* B C | 2 | 150 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 153 | Yes
B | A B* C | 4 | 153 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 160 | Yes
B | [config not available] | | |
C | A* B C | 9 | 160 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 153 | Yes
B | A* B C | 4 | 153 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 8 | 158 | Yes
B | A B* C | 8 | 158 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 144 | Yes
B | A* B C | 2 | 144 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 159 | Yes
B | A B* C D~ | 5 | 159 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 149 | Yes
B | [config not available] | | |
C | A* B C | 7 | 149 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 159 | Yes
B | A B* C D~ | 9 | 159 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 7
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 69
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 40 | Yes
B | [config not available] | | |
C | A B C* | 6 | 40 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 160 | Yes
B | [config not available] | | |
C | A B C* | 5 | 160 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 162 | Yes
B | A B* C D~ | 6 | 162 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 148 | Yes
B | [config not available] | | |
C | A B C* | 4 | 148 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 158 | Yes
B | A B* C D~ | 7 | 158 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 155 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 155 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 8 | 153 | Yes
B | A B* C | 8 | 153 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 156 | Yes
B | A* B C D~ | 2 | 156 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 160 | Yes
B | A B* C D~ | 4 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 160 | Yes
B | [config not available] | | |
C | A* B C | 9 | 160 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 160 | Yes
B | A* B C D~ | 4 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 158 | Yes
B | A B* C | 8 | 158 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 150 | Yes
B | A* B C D~ | 2 | 150 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 159 | Yes
B | A B* C D~ | 5 | 159 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C D~ | 7 | 155 | Yes
B | [config not available] | | |
C | A* B C | 7 | 149 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 9 | 164 | Yes
B | A B* C | 9 | 164 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 166 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 166 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 162 | Yes
B | A B* C D~ | 6 | 162 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 148 | Yes
B | [config not available] | | |
C | A B C* | 4 | 148 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 158 | Yes
B | A B* C D~ | 7 | 158 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 162 | Yes
B | [config not available] | | |
C | A B C* | 7 | 162 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 153 | Yes
B | A B* C | 8 | 153 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 156 | Yes
B | A* B C D~ | 2 | 156 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 160 | Yes
B | A B* C D~ | 4 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 160 | Yes
B | [config not available] | | |
C | A* B C | 9 | 160 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 160 | Yes
B | A* B C D~ | 4 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 158 | Yes
B | A B* C | 8 | 158 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 150 | Yes
B | A* B C D~ | 2 | 150 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 159 | Yes
B | A B* C D~ | 5 | 159 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 155 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 155 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 164 | Yes
B | A B* C | 9 | 164 | Yes
C | [config not available] | | |
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 41 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 41 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 41 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 41 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 166 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 166 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 162 | Yes
B | A B* C D~ | 6 | 162 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 148 | Yes
B | [config not available] | | |
C | A B C* | 4 | 148 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 165 | Yes
B | A B* C | 7 | 165 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 162 | Yes
B | [config not available] | | |
C | A B C* | 7 | 162 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 153 | Yes
B | A B* C | 8 | 153 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 156 | Yes
B | A* B C D~ | 2 | 156 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 160 | Yes
B | A B* C D~ | 4 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 166 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 166 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 160 | Yes
B | A* B C D~ | 4 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 165 | Yes
B | A B* C D~ | 8 | 165 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 150 | Yes
B | A* B C D~ | 2 | 150 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 166 | Yes
B | A B* C | 5 | 166 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 155 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 155 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 164 | Yes
B | A B* C | 9 | 164 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.5:33707, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 41 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 41 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 166 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 166 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 169 | Yes
B | A B* C | 6 | 169 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 155 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 155 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 165 | Yes
B | A B* C | 7 | 165 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 169 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 169 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 160 | Yes
B | A B* C D~ | 8 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 162 | Yes
B | A* B C | 2 | 162 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 165 | Yes
B | A B* C | 4 | 165 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 166 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 166 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 167 | Yes
B | A* B C | 4 | 167 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 165 | Yes
B | A B* C D~ | 8 | 165 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 157 | Yes
B | A* B C | 2 | 157 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 166 | Yes
B | A B* C | 5 | 166 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 161 | Yes
B | [config not available] | | |
C | A* B C | 7 | 161 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 171 | Yes
B | A B* C D~ | 9 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 42 | Yes
B | [config not available] | | |
C | A B C* | 6 | 42 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 172 | Yes
B | [config not available] | | |
C | A B C* | 5 | 172 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 169 | Yes
B | A B* C | 6 | 169 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 155 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 155 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 165 | Yes
B | A B* C | 7 | 165 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 169 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 169 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 160 | Yes
B | A B* C D~ | 8 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 162 | Yes
B | A* B C | 2 | 162 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 165 | Yes
B | A B* C | 4 | 165 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C | 9 | 172 | Yes
B | [config not available] | | |
C | A* B C | 9 | 172 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 167 | Yes
B | A* B C | 4 | 167 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 165 | Yes
B | A B* C D~ | 8 | 165 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 157 | Yes
B | A* B C | 2 | 157 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 166 | Yes
B | A B* C | 5 | 166 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 161 | Yes
B | [config not available] | | |
C | A* B C | 7 | 161 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 171 | Yes
B | A B* C D~ | 9 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 7
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 69
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 42 | Yes
B | [config not available] | | |
C | A B C* | 6 | 42 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 172 | Yes
B | [config not available] | | |
C | A B C* | 5 | 172 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 169 | Yes
B | A B* C | 6 | 169 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 161 | Yes
B | [config not available] | | |
C | A B C* | 4 | 161 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 170 | Yes
B | A B* C D~ | 7 | 170 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 7 | 169 | Yes
B | [config not available] | | |
C | A B C* D~ | 7 | 169 | Yes
D | [config not available] | | |
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 160 | Yes
B | A B* C D~ | 8 | 160 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 162 | Yes
B | A* B C | 2 | 162 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 165 | Yes
B | A B* C | 4 | 165 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 172 | Yes
B | [config not available] | | |
C | A* B C | 9 | 172 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 167 | Yes
B | A* B C | 4 | 167 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 170 | Yes
B | A B* C | 8 | 170 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 163 | Yes
B | A* B C D~ | 2 | 163 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 171 | Yes
B | A B* C D~ | 5 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 161 | Yes
B | [config not available] | | |
C | A* B C | 7 | 161 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 171 | Yes
B | A B* C D~ | 9 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 8
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 69
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.6:44295, tserver@127.22.231.5:33707, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 42 | Yes
B | [config not available] | | |
C | A B C* | 6 | 42 | Yes
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 5 | 172 | Yes
B | [config not available] | | |
C | A B C* | 5 | 172 | Yes
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 174 | Yes
B | A B* C D~ | 6 | 174 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 161 | Yes
B | [config not available] | | |
C | A B C* | 4 | 161 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 170 | Yes
B | A B* C D~ | 7 | 170 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 174 | Yes
B | [config not available] | | |
C | A B C* | 7 | 174 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 165 | Yes
B | A B* C | 8 | 165 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 168 | Yes
B | A* B C D~ | 2 | 168 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 171 | Yes
B | A B* C D~ | 4 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 9 | 172 | Yes
B | [config not available] | | |
C | A* B C | 9 | 172 | Yes
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 174 | Yes
B | A* B C D~ | 4 | 174 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 170 | Yes
B | A B* C | 8 | 170 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 163 | Yes
B | A* B C D~ | 2 | 163 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 171 | Yes
B | A B* C D~ | 5 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 166 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 166 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 176 | Yes
B | A B* C | 9 | 176 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 71
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.2:39099, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 43 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 43 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 177 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 177 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 174 | Yes
B | A B* C D~ | 6 | 174 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 161 | Yes
B | [config not available] | | |
C | A B C* | 4 | 161 | Yes
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 7 | 170 | Yes
B | A B* C D~ | 7 | 170 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 174 | Yes
B | [config not available] | | |
C | A B C* | 7 | 174 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 165 | Yes
B | A B* C | 8 | 165 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 168 | Yes
B | A* B C D~ | 2 | 168 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 171 | Yes
B | A B* C D~ | 4 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 179 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 179 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 174 | Yes
B | A* B C D~ | 4 | 174 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 170 | Yes
B | A B* C | 8 | 170 | Yes
C | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 163 | Yes
B | A* B C D~ | 2 | 163 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 5 | 171 | Yes
B | A B* C D~ | 5 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 166 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 166 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 176 | Yes
B | A B* C | 9 | 176 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 13
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 74
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.3:35521, tserver@127.22.231.6:44295, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 43 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 43 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 177 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 177 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 174 | Yes
B | A B* C D~ | 6 | 174 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 168 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 168 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 177 | Yes
B | A B* C | 7 | 177 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 174 | Yes
B | [config not available] | | |
C | A B C* | 7 | 174 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 165 | Yes
B | A B* C | 8 | 165 | Yes
C | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 168 | Yes
B | A* B C D~ | 2 | 168 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 171 | Yes
B | A B* C D~ | 4 | 171 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 179 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 179 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 174 | Yes
B | A* B C D~ | 4 | 174 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 176 | Yes
B | A B* C D~ | 8 | 176 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 171 | Yes
B | A* B C | 2 | 171 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 177 | Yes
B | A B* C | 5 | 177 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 7 | 166 | Yes
B | [config not available] | | |
C | A* B C D~ | 7 | 166 | Yes
D | [config not available] | | |
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 176 | Yes
B | A B* C | 9 | 176 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 11
Median | 12
Third Quartile | 14
Maximum | 15
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 73
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d0c7c8d1f99e4ccda4d4aa6dd07d886e | 127.22.231.62:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.231.20:33399 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
aff8dec1c558403bb403fe2e08a34305 | 127.22.231.3:35521 | HEALTHY | <none> | 2 | 0
b3ac395387bb4601b607ae4a8404f325 | 127.22.231.6:44295 | HEALTHY | <none> | 3 | 0
c487cdc5c45c4365854a474ec10d6e54 | 127.22.231.2:39099 | HEALTHY | <none> | 7 | 0
f661a2fa62084188b9de0c0e1608bd8b | 127.22.231.5:33707 | HEALTHY | <none> | 9 | 0
12bcc720148d426aa9ca25efe70ac6d4 | 127.22.231.1:42115 | UNAVAILABLE | <none> | n/a | n/a
b6298d218143477eb0770540c0cdbbcc | 127.22.231.4:40461 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.22.231.1:42115: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.1:42115: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.22.231.4:40461: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.22.231.4:40461: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
local_ip_for_outbound_sockets | 127.22.231.2 | experimental | 127.22.231.2:39099
local_ip_for_outbound_sockets | 127.22.231.3 | experimental | 127.22.231.3:35521
local_ip_for_outbound_sockets | 127.22.231.5 | experimental | 127.22.231.5:33707
local_ip_for_outbound_sockets | 127.22.231.6 | experimental | 127.22.231.6:44295
never_fsync | true | unsafe,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_format | pb | hidden | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.231.2:39099
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.231.3:35521
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.231.5:33707
server_dump_info_path | /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.231.6:44295
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.22.231.20:33399 | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
time_source | builtin | 127.22.231.3:35521, 127.22.231.6:44295, 127.22.231.2:39099, and 1 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.22.231.62:45525, tserver@127.22.231.5:33707, tserver@127.22.231.3:35521, and 2 other server(s)
Tablet Summary
Tablet e42879f1c5d04e33a082e6cbd154cabd of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 6 | 43 | Yes
B | [config not available] | | |
C | A B C* D~ | 6 | 43 | Yes
D | [config not available] | | |
Tablet c48d3280db4642f78583237014733ae5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 5 | 177 | Yes
B | [config not available] | | |
C | A B C* D~ | 5 | 177 | Yes
D | [config not available] | | |
Tablet adf37d18ed2a46aba91c451eeae0e1f6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 180 | Yes
B | A B* C | 6 | 180 | Yes
C | [config not available] | | |
Tablet d5fda0ac4aa7417d91de600d14427af6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable [NONVOTER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
D = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 4 | 168 | Yes
B | [config not available] | | |
C | A B C* D~ | 4 | 168 | Yes
D | [config not available] | | |
Tablet 35c85596a2564042a129626f13768a42 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 7 | 177 | Yes
B | A B* C | 7 | 177 | Yes
C | [config not available] | | |
Tablet 0ea0b1495ddb472f9ee9ac06721a6482 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING [LEADER]
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 174 | Yes
B | [config not available] | | |
C | A B C* | 7 | 174 | Yes
Tablet 314ae8aa44184d1da3aaabc0ab29093f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 173 | Yes
B | A B* C D~ | 8 | 173 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 67861a86df1f4536984bc5b6c8eb5df2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 174 | Yes
B | A* B C | 2 | 174 | Yes
C | [config not available] | | |
Tablet 2b272055764c4b7e9eb9703d961e23c4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 176 | Yes
B | A B* C | 4 | 176 | Yes
C | [config not available] | | |
Tablet 6d5cba1382424973929695143efa7e76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = f661a2fa62084188b9de0c0e1608bd8b
B = b6298d218143477eb0770540c0cdbbcc
C = b3ac395387bb4601b607ae4a8404f325
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 9 | 179 | Yes
B | [config not available] | | |
C | A* B C D~ | 9 | 179 | Yes
D | [config not available] | | |
Tablet 31d1169f54f448cb90ce1d6467f5baf8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 181 | Yes
B | A* B C | 4 | 181 | Yes
C | [config not available] | | |
Tablet 01b9a197f5464baf89ec25dd2c8d5867 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 8 | 176 | Yes
B | A B* C D~ | 8 | 176 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet acf5a2b3f9ab411b8c8de017f8329c9f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = aff8dec1c558403bb403fe2e08a34305
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 171 | Yes
B | A* B C | 2 | 171 | Yes
C | [config not available] | | |
Tablet c3fd8f7897174e74aaa841d017103cf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 177 | Yes
B | A B* C | 5 | 177 | Yes
C | [config not available] | | |
Tablet 8e89427782104f9288176ff6726bda56 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
c487cdc5c45c4365854a474ec10d6e54 (127.22.231.2:39099): RUNNING [LEADER]
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable
aff8dec1c558403bb403fe2e08a34305 (127.22.231.3:35521): RUNNING
All reported replicas are:
A = c487cdc5c45c4365854a474ec10d6e54
B = 12bcc720148d426aa9ca25efe70ac6d4
C = aff8dec1c558403bb403fe2e08a34305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 7 | 172 | Yes
B | [config not available] | | |
C | A* B C | 7 | 172 | Yes
Tablet 796121e2021b4939b7310e1216d34f21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b3ac395387bb4601b607ae4a8404f325 (127.22.231.6:44295): RUNNING
f661a2fa62084188b9de0c0e1608bd8b (127.22.231.5:33707): RUNNING [LEADER]
b6298d218143477eb0770540c0cdbbcc (127.22.231.4:40461): TS unavailable
12bcc720148d426aa9ca25efe70ac6d4 (127.22.231.1:42115): TS unavailable [NONVOTER]
All reported replicas are:
A = b3ac395387bb4601b607ae4a8404f325
B = f661a2fa62084188b9de0c0e1608bd8b
C = b6298d218143477eb0770540c0cdbbcc
D = 12bcc720148d426aa9ca25efe70ac6d4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 9 | 182 | Yes
B | A B* C D~ | 9 | 182 | Yes
C | [config not available] | | |
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 5 | 0 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 10
Median | 12
Third Quartile | 14
Maximum | 14
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 70
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
E20240501 20:13:35.255959 23452 external_mini_cluster.cc:943] Process with UUID 12bcc720148d426aa9ca25efe70ac6d4 has crashed
E20240501 20:13:35.256105 23452 external_mini_cluster.cc:943] Process with UUID b6298d218143477eb0770540c0cdbbcc has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240501 20:13:35.938659 23452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu pid 23684
I20240501 20:13:36.296001 23452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu with pid 23684
I20240501 20:13:36.346738 23452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu pid 23815
I20240501 20:13:36.702172 23452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu with pid 23815
I20240501 20:13:36.752406 23452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu pid 24075
I20240501 20:13:37.098722 23452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu with pid 24075
I20240501 20:13:37.147287 23452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu pid 24205
I20240501 20:13:37.471475 23452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu with pid 24205
I20240501 20:13:37.514596 23452 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu pid 23464
I20240501 20:13:37.788246 23452 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskj8Q6Xt/build/asan/bin/kudu with pid 23464
2024-05-01T20:13:37Z chronyd exiting
I20240501 20:13:37.849462 23452 test_util.cc:175] -----------------------------------------------
I20240501 20:13:37.849548 23452 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskj8Q6Xt/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714594305720801-23452-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (112066 ms)
[----------] 1 test from RaftConsensusStressITest (112066 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (112066 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240501 20:13:37.850714 23452 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 9 messages since previous log ~39 seconds ago