[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:11:55.260445 12715 test_util.cc:268] Using random seed: 1297168883
I20240503 14:11:55.271725 12715 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:11:55.271806 12715 ts_itest-base.cc:113] --------------
I20240503 14:11:55.271847 12715 ts_itest-base.cc:114] 6 tablet servers
I20240503 14:11:55.271881 12715 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:11:55.271919 12715 ts_itest-base.cc:116] --------------
2024-05-03T14:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:11:55Z Disabled control of system clock
I20240503 14:11:55.295586 12715 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
/tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.106.254:43013
--webserver_interface=127.12.106.254
--webserver_port=0
--builtin_ntp_servers=127.12.106.212:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.106.254:43013
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=20 with env {}
I20240503 14:11:55.732390 12715 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu as pid 12727
I20240503 14:11:55.732676 12715 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:11:55.849247 12715 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
/tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.106.193:0
--local_ip_for_outbound_sockets=127.12.106.193
--webserver_interface=127.12.106.193
--webserver_port=0
--tserver_master_addrs=127.12.106.254:43013
--builtin_ntp_servers=127.12.106.212:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240503 14:11:56.357642 12715 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu as pid 12817
I20240503 14:11:56.357883 12715 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:11:56.361512 12715 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
/tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.106.194:0
--local_ip_for_outbound_sockets=127.12.106.194
--webserver_interface=127.12.106.194
--webserver_port=0
--tserver_master_addrs=127.12.106.254:43013
--builtin_ntp_servers=127.12.106.212:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240503 14:11:56.810565 12715 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu as pid 12948
I20240503 14:11:56.810768 12715 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:11:56.814520 12715 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
/tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.106.195:0
--local_ip_for_outbound_sockets=127.12.106.195
--webserver_interface=127.12.106.195
--webserver_port=0
--tserver_master_addrs=127.12.106.254:43013
--builtin_ntp_servers=127.12.106.212:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240503 14:11:57.276643 12715 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu as pid 13078
I20240503 14:11:57.276814 12715 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:11:57.280462 12715 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
/tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.106.196:0
--local_ip_for_outbound_sockets=127.12.106.196
--webserver_interface=127.12.106.196
--webserver_port=0
--tserver_master_addrs=127.12.106.254:43013
--builtin_ntp_servers=127.12.106.212:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240503 14:11:57.748322 12715 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu as pid 13208
I20240503 14:11:57.748505 12715 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:11:57.752455 12715 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
/tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.106.197:0
--local_ip_for_outbound_sockets=127.12.106.197
--webserver_interface=127.12.106.197
--webserver_port=0
--tserver_master_addrs=127.12.106.254:43013
--builtin_ntp_servers=127.12.106.212:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240503 14:11:58.201570 12715 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu as pid 13338
I20240503 14:11:58.201766 12715 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:11:58.204725 12715 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
/tmp/dist-test-tasks6x_j7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.106.198:0
--local_ip_for_outbound_sockets=127.12.106.198
--webserver_interface=127.12.106.198
--webserver_port=0
--tserver_master_addrs=127.12.106.254:43013
--builtin_ntp_servers=127.12.106.212:39781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240503 14:11:58.671805 12715 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu as pid 13468
I20240503 14:11:58.672003 12715 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:11:58.690750 12715 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 14:11:58.978252 12715 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 14:11:58.981503 12715 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 988abd63b29e48aaa1379a40430c3d97 to finish bootstrapping
I20240503 14:11:58.990973 12715 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ce4502e3ac5d40d09cfa8980f8ae9e2b to finish bootstrapping
I20240503 14:11:59.000541 12715 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3df896b75e6d46b0ab26ccf5226ec5da to finish bootstrapping
I20240503 14:11:59.007702 12715 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4aa808b5aaaf4b15907d66664a6a47b5 to finish bootstrapping
I20240503 14:11:59.014647 12715 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 89ffc01736ad498bb565d0beb560c96c to finish bootstrapping
I20240503 14:11:59.023092 12715 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 168f1f18074549f68beff4deb648c2cb to finish bootstrapping
I20240503 14:11:59.031608 12715 test_util.cc:268] Using random seed: 1300940045
W20240503 14:12:07.680532 13950 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:12:04.451036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.106.197:40641 (local address 127.0.0.1:35262)
0503 14:12:04.451269 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0503 14:12:04.451288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:12:04.451394 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:12:07.679178 (+3227784us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.12.106.197:40641: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":109}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 4 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 1 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | HEALTHY | <none> | 8 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 6 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 2 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.197:40641: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.197:40641: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.196:38389, and 3 other server(s)
Tablet Summary
Tablet 5999cbd2a5634b40aae7a643aa1d82dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING [LEADER]
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING
All reported replicas are:
A = 89ffc01736ad498bb565d0beb560c96c
B = 168f1f18074549f68beff4deb648c2cb
C = ce4502e3ac5d40d09cfa8980f8ae9e2b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 9d313431c2c64447b5b5f9bd4d0867de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING [LEADER]
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 0947260516b6466ca30087cb569a8c3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet e90632df9949419091fbc2faa48d8d78 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
Tablet 17ef3e3ace68469188b7902d9a9f6c64 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING [LEADER]
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet ef66615f0771496584b0330e8c807024 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 8d9716ef10c04e5193a5276675f6292c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 616c11354f784b7ab2b195df166654ec of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet c3769db4162e46c5bbf4bda4232277a2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING [LEADER]
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet b474b90e59254fab8a5fc6b4cf693f6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 89ffc01736ad498bb565d0beb560c96c
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 10 | 0 | 10 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 11
Third Quartile | 11
Maximum | 11
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 4 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 1 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | HEALTHY | <none> | 7 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 0 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 6 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 11
Third Quartile | 11
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:12:12.445654 12715 raft_consensus_stress-itest.cc:258] completed iteration 0
W20240503 14:12:27.395380 14870 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:12:24.215733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.106.197:40641 (local address 127.0.0.1:42984)
0503 14:12:24.217513 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0503 14:12:24.217532 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:12:24.217630 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:12:27.395130 (+3177500us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.12.106.197:40641: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":53}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 5 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | HEALTHY | <none> | 6 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 10 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.197:40641: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.197:40641: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
Tablet 42555d11957e472cb9c67bc713b749bf of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 4 | Yes
B | A B* C | 1 | 4 | Yes
C | [config not available] | | |
Tablet 3d65c6077a8748dcac0a53da77fa85ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 51 | Yes
B | A* B C | 2 | 51 | Yes
C | [config not available] | | |
Tablet d24765a6c53b44c2ae6c161b4c74c7ab of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 50 | Yes
B | A* B C | 1 | 50 | Yes
C | [config not available] | | |
Tablet ad58c3a2c2e24398837cf8bd52e9b8df of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 49 | Yes
B | A* B C | 1 | 49 | Yes
C | [config not available] | | |
Tablet 2b70f77405634a9a9f42c405c94a911d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 51 | Yes
B | A* B C | 1 | 51 | Yes
C | [config not available] | | |
Tablet 72dd5fc3733b4121ac3a223e2716be1e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 51 | Yes
B | A* B C | 1 | 51 | Yes
C | [config not available] | | |
Tablet edaf092a0ada466dba2df44140e8bdf1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 53 | Yes
B | A* B C | 2 | 53 | Yes
C | [config not available] | | |
Tablet c1f0736b472945a994bf44469859b939 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 53 | Yes
B | A* B C | 1 | 53 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 13 | 0 | 7 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 2
Median | 10
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 6 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 1 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | HEALTHY | <none> | 5 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 0 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 8 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 17
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:12:32.296649 12715 raft_consensus_stress-itest.cc:258] completed iteration 1
W20240503 14:12:40.959301 15435 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:12:37.924725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.106.198:39613 (local address 127.0.0.1:47246)
0503 14:12:37.924951 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0503 14:12:37.924975 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0503 14:12:37.925089 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:12:40.959093 (+3034004us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.12.106.198:39613: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":80}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 1 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | HEALTHY | <none> | 5 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 0 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 9 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 0 | 0
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.198:39613: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.198:39613: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.196:38389, and 3 other server(s)
Tablet Summary
Tablet 42555d11957e472cb9c67bc713b749bf of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 10 | Yes
B | A B* C | 1 | 10 | Yes
C | [config not available] | | |
Tablet 5999cbd2a5634b40aae7a643aa1d82dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable [LEADER]
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
All reported replicas are:
A = 168f1f18074549f68beff4deb648c2cb
B = 988abd63b29e48aaa1379a40430c3d97
C = 4aa808b5aaaf4b15907d66664a6a47b5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | 52 | Yes
C | A* B C | 1 | 52 | Yes
Tablet 3d65c6077a8748dcac0a53da77fa85ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 54 | Yes
B | A* B C | 2 | 54 | Yes
C | [config not available] | | |
Tablet 9d313431c2c64447b5b5f9bd4d0867de of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable [LEADER]
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 40 | Yes
B | [config not available] | | |
C | A B* C | 1 | 40 | Yes
Tablet d24765a6c53b44c2ae6c161b4c74c7ab of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 53 | Yes
B | A* B C | 1 | 53 | Yes
C | [config not available] | | |
Tablet 0947260516b6466ca30087cb569a8c3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 42 | Yes
B | [config not available] | | |
C | A* B C | 1 | 42 | Yes
Tablet ad58c3a2c2e24398837cf8bd52e9b8df of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 52 | Yes
B | A* B C | 1 | 52 | Yes
C | [config not available] | | |
Tablet e90632df9949419091fbc2faa48d8d78 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable [LEADER]
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 44 | Yes
B | [config not available] | | |
C | A B* C | 3 | 44 | Yes
Tablet 17ef3e3ace68469188b7902d9a9f6c64 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable [LEADER]
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 44 | Yes
B | [config not available] | | |
C | A B* C | 3 | 44 | Yes
Tablet ef66615f0771496584b0330e8c807024 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 46 | Yes
B | [config not available] | | |
C | A* B C | 1 | 46 | Yes
Tablet 2b70f77405634a9a9f42c405c94a911d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 54 | Yes
B | A* B C | 1 | 54 | Yes
C | [config not available] | | |
Tablet 8d9716ef10c04e5193a5276675f6292c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 43 | Yes
B | [config not available] | | |
C | A* B C | 1 | 43 | Yes
Tablet 72dd5fc3733b4121ac3a223e2716be1e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 54 | Yes
B | A* B C | 1 | 54 | Yes
C | [config not available] | | |
Tablet 616c11354f784b7ab2b195df166654ec of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable [LEADER]
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 46 | Yes
B | [config not available] | | |
C | A B* C | 2 | 46 | Yes
Tablet edaf092a0ada466dba2df44140e8bdf1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 56 | Yes
B | A* B C | 2 | 56 | Yes
C | [config not available] | | |
Tablet c3769db4162e46c5bbf4bda4232277a2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable [LEADER]
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 39 | Yes
B | [config not available] | | |
C | A B* C | 1 | 39 | Yes
Tablet c1f0736b472945a994bf44469859b939 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 168f1f18074549f68beff4deb648c2cb
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 56 | Yes
B | A* B C | 1 | 56 | Yes
C | [config not available] | | |
Tablet b474b90e59254fab8a5fc6b4cf693f6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING [LEADER]
168f1f18074549f68beff4deb648c2cb (127.12.106.198:39613): TS unavailable
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 168f1f18074549f68beff4deb648c2cb
C = 988abd63b29e48aaa1379a40430c3d97
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 44 | Yes
B | [config not available] | | |
C | A* B C | 1 | 44 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 3 | 0 | 17 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 18
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240503 14:12:48.127372 15795 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:12:44.872043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.106.197:40641 (local address 127.0.0.1:46672)
0503 14:12:44.872723 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0503 14:12:44.872741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 14:12:44.872858 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:12:48.127157 (+3254299us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.12.106.197:40641: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":546}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 0 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 1 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | HEALTHY | <none> | 10 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 10 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.197:40641: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.197:40641: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
Tablet d1f9a4ea97c34455b705d0f4558a5516 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 54 | Yes
B | A* B C | 4 | 54 | Yes
C | [config not available] | | |
Tablet 3d65c6077a8748dcac0a53da77fa85ae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 61 | Yes
B | A* B C | 2 | 61 | Yes
C | [config not available] | | |
Tablet d24765a6c53b44c2ae6c161b4c74c7ab of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 60 | Yes
B | A* B C | 1 | 60 | Yes
C | [config not available] | | |
Tablet ad58c3a2c2e24398837cf8bd52e9b8df of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 59 | Yes
B | A* B C | 1 | 59 | Yes
C | [config not available] | | |
Tablet 565aa8ee57c84c0a87b1f92ce4750f88 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 55 | Yes
B | A* B C | 2 | 55 | Yes
C | [config not available] | | |
Tablet 2b70f77405634a9a9f42c405c94a911d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 60 | Yes
B | A* B C | 1 | 60 | Yes
C | [config not available] | | |
Tablet 72dd5fc3733b4121ac3a223e2716be1e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 61 | Yes
B | A* B C | 1 | 61 | Yes
C | [config not available] | | |
Tablet edaf092a0ada466dba2df44140e8bdf1 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 63 | Yes
B | A* B C | 2 | 63 | Yes
C | [config not available] | | |
Tablet c1f0736b472945a994bf44469859b939 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): RUNNING
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 4aa808b5aaaf4b15907d66664a6a47b5
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 63 | Yes
B | A* B C | 1 | 63 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 11 | 0 | 9 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 2
Median | 9
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 0 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 1 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | HEALTHY | <none> | 10 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 0 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 10 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 10
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:12:52.281692 12715 raft_consensus_stress-itest.cc:258] completed iteration 2
W20240503 14:13:07.839521 16764 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:13:04.595545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.106.196:38389 (local address 127.0.0.1:42370)
0503 14:13:04.596799 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0503 14:13:04.596820 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0503 14:13:04.596961 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:13:07.839142 (+3242181us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.12.106.196:38389: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":2083,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 2 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 5 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 0 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 0 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
Tablet 0947260516b6466ca30087cb569a8c3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): TS unavailable [LEADER]
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 988abd63b29e48aaa1379a40430c3d97
C = ce4502e3ac5d40d09cfa8980f8ae9e2b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 5 | 53 | Yes
C | A* B C | 5 | 53 | Yes
Tablet e90632df9949419091fbc2faa48d8d78 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = 4aa808b5aaaf4b15907d66664a6a47b5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 5 | 69 | Yes
B | A* B C | 5 | 69 | Yes
C | [config not available] | | |
Tablet ef66615f0771496584b0330e8c807024 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = 4aa808b5aaaf4b15907d66664a6a47b5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 69 | Yes
B | A* B C | 2 | 69 | Yes
C | [config not available] | | |
Tablet 8d9716ef10c04e5193a5276675f6292c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = 4aa808b5aaaf4b15907d66664a6a47b5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 64 | Yes
B | A* B C | 2 | 64 | Yes
C | [config not available] | | |
Tablet 616c11354f784b7ab2b195df166654ec of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): TS unavailable
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): RUNNING [LEADER]
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 988abd63b29e48aaa1379a40430c3d97
C = 3df896b75e6d46b0ab26ccf5226ec5da
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 5 | 66 | Yes
C | A B C* | 5 | 66 | Yes
Tablet c3769db4162e46c5bbf4bda4232277a2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): RUNNING [LEADER]
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = 4aa808b5aaaf4b15907d66664a6a47b5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 63 | Yes
B | A B* C | 6 | 63 | Yes
C | [config not available] | | |
Tablet b474b90e59254fab8a5fc6b4cf693f6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389): TS unavailable
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): RUNNING [LEADER]
All reported replicas are:
A = 4aa808b5aaaf4b15907d66664a6a47b5
B = 988abd63b29e48aaa1379a40430c3d97
C = 3df896b75e6d46b0ab26ccf5226ec5da
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 3 | 63 | Yes
C | A B C* | 3 | 63 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 13 | 0 | 7 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 7
Median | 11
Third Quartile | 13
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 2 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 0 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 14 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.196 | experimental | 127.12.106.196:38389
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.12.106.196:38389
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 7
Median | 11
Third Quartile | 13
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:13:12.149804 12715 raft_consensus_stress-itest.cc:258] completed iteration 3
F20240503 14:13:15.679870 17166 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0947260516b6466ca30087cb569a8c3d.queue_time_us, tablet-download-pool-0947260516b6466ca30087cb569a8c3d.queue_time_us) (tablet-download-pool-2b70f77405634a9a9f42c405c94a911d.queue_time_us, tablet-download-pool-2b70f77405634a9a9f42c405c94a911d.queue_time_us) (tablet-download-pool-2b70f77405634a9a9f42c405c94a911d.run_cpu_time_us, tablet-download-pool-2b70f77405634a9a9f42c405c94a911d.run_cpu_time_us) (tablet-download-pool-2b70f77405634a9a9f42c405c94a911d.run_wall_time_us, tablet-download-pool-2b70f77405634a9a9f42c405c94a911d.run_wall_time_us) (tablet-download-pool-3d65c6077a8748dcac0a53da77fa85ae.queue_time_us, tablet-download-pool-3d65c6077a8748dcac0a53da77fa85ae.queue_time_us) (tablet-download-pool-3d65c6077a8748dcac0a53da77fa85ae.run_cpu_time_us, tablet-download-pool-3d65c6077a8748dcac0a53da77fa85ae.run_cpu_time_us) (tablet-download-pool-3d65c6077a8748dcac0a53da77fa85ae.run_wall_time_us, tablet-download-pool-3d65c6077a8748dcac0a53da77fa85ae.run_wall_time_us) (tablet-download-pool-565aa8ee57c84c0a87b1f92ce4750f88.queue_time_us, tablet-download-pool-565aa8ee57c84c0a87b1f92ce4750f88.queue_time_us) (tablet-download-pool-565aa8ee57c84c0a87b1f92ce4750f88.run_cpu_time_us, tablet-download-pool-565aa8ee57c84c0a87b1f92ce4750f88.run_cpu_time_us) (tablet-download-pool-565aa8ee57c84c0a87b1f92ce4750f88.run_wall_time_us, tablet-download-pool-565aa8ee57c84c0a87b1f92ce4750f88.run_wall_time_us) (tablet-download-pool-5999cbd2a5634b40aae7a643aa1d82dd.queue_time_us, tablet-download-pool-5999cbd2a5634b40aae7a643aa1d82dd.queue_time_us) (tablet-download-pool-5999cbd2a5634b40aae7a643aa1d82dd.run_cpu_time_us, tablet-download-pool-5999cbd2a5634b40aae7a643aa1d82dd.run_cpu_time_us) (tablet-download-pool-5999cbd2a5634b40aae7a643aa1d82dd.run_wall_time_us, tablet-download-pool-5999cbd2a5634b40aae7a643aa1d82dd.run_wall_time_us) (tablet-download-pool-72dd5fc3733b4121ac3a223e2716be1e.queue_time_us, tablet-download-pool-72dd5fc3733b4121ac3a223e2716be1e.queue_time_us) (tablet-download-pool-72dd5fc3733b4121ac3a223e2716be1e.run_cpu_time_us, tablet-download-pool-72dd5fc3733b4121ac3a223e2716be1e.run_cpu_time_us) (tablet-download-pool-72dd5fc3733b4121ac3a223e2716be1e.run_wall_time_us, tablet-download-pool-72dd5fc3733b4121ac3a223e2716be1e.run_wall_time_us) (tablet-download-pool-8d9716ef10c04e5193a5276675f6292c.queue_time_us, tablet-download-pool-8d9716ef10c04e5193a5276675f6292c.queue_time_us) (tablet-download-pool-8d9716ef10c04e5193a5276675f6292c.run_cpu_time_us, tablet-download-pool-8d9716ef10c04e5193a5276675f6292c.run_cpu_time_us) (tablet-download-pool-8d9716ef10c04e5193a5276675f6292c.run_wall_time_us, tablet-download-pool-8d9716ef10c04e5193a5276675f6292c.run_wall_time_us) (tablet-download-pool-ad58c3a2c2e24398837cf8bd52e9b8df.queue_time_us, tablet-download-pool-ad58c3a2c2e24398837cf8bd52e9b8df.queue_time_us) (tablet-download-pool-ad58c3a2c2e24398837cf8bd52e9b8df.run_cpu_time_us, tablet-download-pool-ad58c3a2c2e24398837cf8bd52e9b8df.run_cpu_time_us) (tablet-download-pool-ad58c3a2c2e24398837cf8bd52e9b8df.run_wall_time_us, tablet-download-pool-ad58c3a2c2e24398837cf8bd52e9b8df.run_wall_time_us) (tablet-download-pool-b8af0069fec0460ca1ada8b490cebaf2.queue_time_us, tablet-download-pool-b8af0069fec0460ca1ada8b490cebaf2.queue_time_us) (tablet-download-pool-b8af0069fec0460ca1ada8b490cebaf2.run_cpu_time_us, tablet-download-pool-b8af0069fec0460ca1ada8b490cebaf2.run_cpu_time_us) (tablet-download-pool-b8af0069fec0460ca1ada8b490cebaf2.run_wall_time_us, tablet-download-pool-b8af0069fec0460ca1ada8b490cebaf2.run_wall_time_us) (tablet-download-pool-c1f0736b472945a994bf44469859b939.queue_time_us, tablet-download-pool-c1f0736b472945a994bf44469859b939.queue_time_us) (tablet-download-pool-c1f0736b472945a994bf44469859b939.run_cpu_time_us, tablet-download-pool-c1f0736b472945a994bf44469859b939.run_cpu_time_us) (tablet-download-pool-c1f0736b472945a994bf44469859b939.run_wall_time_us, tablet-download-pool-c1f0736b472945a994bf44469859b939.run_wall_time_us) (tablet-download-pool-c3769db4162e46c5bbf4bda4232277a2.queue_time_us, tablet-download-pool-c3769db4162e46c5bbf4bda4232277a2.queue_time_us) (tablet-download-pool-c3769db4162e46c5bbf4bda4232277a2.run_cpu_time_us, tablet-download-pool-c3769db4162e46c5bbf4bda4232277a2.run_cpu_time_us) (tablet-download-pool-c3769db4162e46c5bbf4bda4232277a2.run_wall_time_us, tablet-download-pool-c3769db4162e46c5bbf4bda4232277a2.run_wall_time_us) (tablet-download-pool-d24765a6c53b44c2ae6c161b4c74c7ab.queue_time_us, tablet-download-pool-d24765a6c53b44c2ae6c161b4c74c7ab.queue_time_us) (tablet-download-pool-d24765a6c53b44c2ae6c161b4c74c7ab.run_cpu_time_us, tablet-download-pool-d24765a6c53b44c2ae6c161b4c74c7ab.run_cpu_time_us) (tablet-download-pool-d24765a6c53b44c2ae6c161b4c74c7ab.run_wall_time_us, tablet-download-pool-d24765a6c53b44c2ae6c161b4c74c7ab.run_wall_time_us) (tablet-download-pool-e90632df9949419091fbc2faa48d8d78.queue_time_us, tablet-download-pool-e90632df9949419091fbc2faa48d8d78.queue_time_us) (tablet-download-pool-e90632df9949419091fbc2faa48d8d78.run_cpu_time_us, tablet-download-pool-e90632df9949419091fbc2faa48d8d78.run_cpu_time_us) (tablet-download-pool-e90632df9949419091fbc2faa48d8d78.run_wall_time_us, tablet-download-pool-e90632df9949419091fbc2faa48d8d78.run_wall_time_us) (tablet-download-pool-edaf092a0ada466dba2df44140e8bdf1.queue_time_us, tablet-download-pool-edaf092a0ada466dba2df44140e8bdf1.queue_time_us) (tablet-download-pool-edaf092a0ada466dba2df44140e8bdf1.run_cpu_time_us, tablet-download-pool-edaf092a0ada466dba2df44140e8bdf1.run_cpu_time_us) (tablet-download-pool-edaf092a0ada466dba2df44140e8bdf1.run_wall_time_us, tablet-download-pool-edaf092a0ada466dba2df44140e8bdf1.run_wall_time_us) (tablet-download-pool-ef66615f0771496584b0330e8c807024.queue_time_us, tablet-download-pool-ef66615f0771496584b0330e8c807024.queue_time_us) (tablet-download-pool-ef66615f0771496584b0330e8c807024.run_cpu_time_us, tablet-download-pool-ef66615f0771496584b0330e8c807024.run_cpu_time_us) (tablet-download-pool-ef66615f0771496584b0330e8c807024.run_wall_time_us, tablet-download-pool-ef66615f0771496584b0330e8c807024.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f16bc304dcd google::LogMessage::Fail() at ??:0
@ 0x7f16bc308b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f16bc3047cc google::LogMessage::Flush() at ??:0
@ 0x7f16bc305f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f16bf045742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f16bf011f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f16bf00bcd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f16ce45f8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f16ce477daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f16ce5a2480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f16ce5daa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f16ce5bd981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f16ce5bd70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f16bf015b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f16bf01919c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f16bf019019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f16beff401e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f16c23796db start_thread at ??:0
@ 0x7f16bb1b771f clone at ??:0
W20240503 14:13:17.397617 13633 meta_cache.cc:302] tablet b474b90e59254fab8a5fc6b4cf693f6e: replica 4aa808b5aaaf4b15907d66664a6a47b5 (127.12.106.196:38389) has failed: Network error: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111)
W20240503 14:13:27.295491 17493 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:13:24.232750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.106.195:33321 (local address 127.0.0.1:36746)
0503 14:13:24.244637 (+ 11887us) negotiation.cc:107] Waiting for socket to connect
0503 14:13:24.244659 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0503 14:13:24.244797 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:13:27.295164 (+3050367us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.12.106.195:33321: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":11778,"thread_start_us":13768,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 1 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 15 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 4 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | UNAVAILABLE | <none> | n/a | n/a
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.195:33321: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.195:33321: received 0 of 4 requested bytes (UNAVAILABLE)
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.197:40641, and 2 other server(s)
Tablet Summary
Tablet 5999cbd2a5634b40aae7a643aa1d82dd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): RUNNING
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 6 | 77 | Yes
B | [config not available] | | |
C | A* B C | 6 | 77 | Yes
Tablet 0947260516b6466ca30087cb569a8c3d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = ce4502e3ac5d40d09cfa8980f8ae9e2b
C = 3df896b75e6d46b0ab26ccf5226ec5da
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 6 | 71 | Yes
B | A* B C | 6 | 71 | Yes
C | [config not available] | | |
Tablet e90632df9949419091fbc2faa48d8d78 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = ce4502e3ac5d40d09cfa8980f8ae9e2b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 5 | 78 | Yes
B | [config not available] | | |
C | A* B C | 5 | 78 | Yes
Tablet 17ef3e3ace68469188b7902d9a9f6c64 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = ce4502e3ac5d40d09cfa8980f8ae9e2b
C = 3df896b75e6d46b0ab26ccf5226ec5da
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 8 | 79 | Yes
B | A* B C | 8 | 79 | Yes
C | [config not available] | | |
Tablet b8af0069fec0460ca1ada8b490cebaf2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable [LEADER]
89ffc01736ad498bb565d0beb560c96c (127.12.106.197:40641): RUNNING
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = 89ffc01736ad498bb565d0beb560c96c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 8 | 72 | Yes
B | [config not available] | | |
C | A B* C | 8 | 72 | Yes
Tablet ef66615f0771496584b0330e8c807024 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING [LEADER]
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = ce4502e3ac5d40d09cfa8980f8ae9e2b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 79 | Yes
B | [config not available] | | |
C | A B C* | 3 | 79 | Yes
Tablet 8d9716ef10c04e5193a5276675f6292c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING [LEADER]
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = ce4502e3ac5d40d09cfa8980f8ae9e2b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 74 | Yes
B | [config not available] | | |
C | A* B C | 2 | 74 | Yes
Tablet 616c11354f784b7ab2b195df166654ec of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING [LEADER]
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = ce4502e3ac5d40d09cfa8980f8ae9e2b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 77 | Yes
B | [config not available] | | |
C | A B C* | 6 | 77 | Yes
Tablet c3769db4162e46c5bbf4bda4232277a2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING [LEADER]
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = ce4502e3ac5d40d09cfa8980f8ae9e2b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 7 | 73 | Yes
B | [config not available] | | |
C | A B C* | 7 | 73 | Yes
Tablet b474b90e59254fab8a5fc6b4cf693f6e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
988abd63b29e48aaa1379a40430c3d97 (127.12.106.193:39129): RUNNING
3df896b75e6d46b0ab26ccf5226ec5da (127.12.106.195:33321): TS unavailable
ce4502e3ac5d40d09cfa8980f8ae9e2b (127.12.106.194:37901): RUNNING [LEADER]
All reported replicas are:
A = 988abd63b29e48aaa1379a40430c3d97
B = 3df896b75e6d46b0ab26ccf5226ec5da
C = ce4502e3ac5d40d09cfa8980f8ae9e2b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 74 | Yes
B | [config not available] | | |
C | A B C* | 4 | 74 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 10 | 0 | 10 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.194:37901, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.193:39129, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.193:39129, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.197:40641, tserver@127.12.106.194:37901, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.193:39129, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.197:40641, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.195:33321, tserver@127.12.106.198:39613, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.197:40641, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.194:37901, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c71611e495c14889a1a2d69c919af545 | 127.12.106.254:43013 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.106.212:39781 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
168f1f18074549f68beff4deb648c2cb | 127.12.106.198:39613 | HEALTHY | <none> | 1 | 0
3df896b75e6d46b0ab26ccf5226ec5da | 127.12.106.195:33321 | HEALTHY | <none> | 0 | 0
89ffc01736ad498bb565d0beb560c96c | 127.12.106.197:40641 | HEALTHY | <none> | 2 | 0
988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | HEALTHY | <none> | 13 | 0
ce4502e3ac5d40d09cfa8980f8ae9e2b | 127.12.106.194:37901 | HEALTHY | <none> | 5 | 0
4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.12.106.196:38389: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.12.106.196:38389: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
local_ip_for_outbound_sockets | 127.12.106.193 | experimental | 127.12.106.193:39129
local_ip_for_outbound_sockets | 127.12.106.194 | experimental | 127.12.106.194:37901
local_ip_for_outbound_sockets | 127.12.106.195 | experimental | 127.12.106.195:33321
local_ip_for_outbound_sockets | 127.12.106.197 | experimental | 127.12.106.197:40641
local_ip_for_outbound_sockets | 127.12.106.198 | experimental | 127.12.106.198:39613
never_fsync | true | unsafe,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.12.106.193:39129
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.12.106.194:37901
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.12.106.195:33321
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.12.106.197:40641
server_dump_info_path | /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.12.106.198:39613
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-----------------------------------------------------------------------------------------
builtin_ntp_servers | 127.12.106.212:39781 | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
time_source | builtin | 127.12.106.198:39613, 127.12.106.195:33321, 127.12.106.197:40641, and 2 other server(s)
Version Summary
Version | Servers
-----------------+----------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.12.106.254:43013, tserver@127.12.106.198:39613, tserver@127.12.106.195:33321, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 13
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Big | 988abd63b29e48aaa1379a40430c3d97 | 127.12.106.193:39129 | 21
Small | 4aa808b5aaaf4b15907d66664a6a47b5 | 127.12.106.196:38389 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
E20240503 14:14:10.707492 12715 external_mini_cluster.cc:943] Process with UUID 4aa808b5aaaf4b15907d66664a6a47b5 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240503 14:14:12.600592 12715 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu pid 12817
I20240503 14:14:12.962297 12715 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu with pid 12817
I20240503 14:14:13.023380 12715 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu pid 12948
I20240503 14:14:13.359154 12715 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu with pid 12948
I20240503 14:14:13.408077 12715 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu pid 13078
I20240503 14:14:13.734028 12715 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu with pid 13078
I20240503 14:14:13.802320 12715 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu pid 13338
I20240503 14:14:14.101106 12715 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu with pid 13338
I20240503 14:14:14.152303 12715 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu pid 13468
I20240503 14:14:14.443181 12715 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu with pid 13468
I20240503 14:14:14.491719 12715 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu pid 12727
I20240503 14:14:14.761704 12715 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks6x_j7/build/asan/bin/kudu with pid 12727
2024-05-03T14:14:14Z chronyd exiting
I20240503 14:14:14.822387 12715 test_util.cc:175] -----------------------------------------------
I20240503 14:14:14.822468 12715 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasks6x_j7/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714745515190178-12715-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (139566 ms)
[----------] 1 test from RaftConsensusStressITest (139566 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (139566 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240503 14:14:14.823477 12715 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~57 seconds ago