[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:55:21.113073 18884 test_util.cc:268] Using random seed: -447959622
I20240502 01:55:21.125306 18884 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:55:21.125398 18884 ts_itest-base.cc:113] --------------
I20240502 01:55:21.125445 18884 ts_itest-base.cc:114] 6 tablet servers
I20240502 01:55:21.125506 18884 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:55:21.125571 18884 ts_itest-base.cc:116] --------------
2024-05-02T01:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:55:21Z Disabled control of system clock
I20240502 01:55:21.152406 18884 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
/tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.113.62:38409
--webserver_interface=127.18.113.62
--webserver_port=0
--builtin_ntp_servers=127.18.113.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.113.62:38409
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=20 with env {}
I20240502 01:55:21.622277 18884 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu as pid 18896
I20240502 01:55:21.622534 18884 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:55:21.718211 18884 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
/tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.113.1:0
--local_ip_for_outbound_sockets=127.18.113.1
--webserver_interface=127.18.113.1
--webserver_port=0
--tserver_master_addrs=127.18.113.62:38409
--builtin_ntp_servers=127.18.113.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 01:55:22.189363 18884 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu as pid 18986
I20240502 01:55:22.189608 18884 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:55:22.193302 18884 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
/tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.113.2:0
--local_ip_for_outbound_sockets=127.18.113.2
--webserver_interface=127.18.113.2
--webserver_port=0
--tserver_master_addrs=127.18.113.62:38409
--builtin_ntp_servers=127.18.113.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 01:55:22.663044 18884 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu as pid 19117
I20240502 01:55:22.663214 18884 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:55:22.667064 18884 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
/tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.113.3:0
--local_ip_for_outbound_sockets=127.18.113.3
--webserver_interface=127.18.113.3
--webserver_port=0
--tserver_master_addrs=127.18.113.62:38409
--builtin_ntp_servers=127.18.113.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 01:55:23.136844 18884 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu as pid 19247
I20240502 01:55:23.137074 18884 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:55:23.140833 18884 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
/tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.113.4:0
--local_ip_for_outbound_sockets=127.18.113.4
--webserver_interface=127.18.113.4
--webserver_port=0
--tserver_master_addrs=127.18.113.62:38409
--builtin_ntp_servers=127.18.113.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 01:55:23.610639 18884 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu as pid 19377
I20240502 01:55:23.610872 18884 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:55:23.614308 18884 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
/tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.113.5:0
--local_ip_for_outbound_sockets=127.18.113.5
--webserver_interface=127.18.113.5
--webserver_port=0
--tserver_master_addrs=127.18.113.62:38409
--builtin_ntp_servers=127.18.113.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 01:55:24.106956 18884 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu as pid 19507
I20240502 01:55:24.107187 18884 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:55:24.111322 18884 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
/tmp/dist-test-tasksN0TNC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.113.6:0
--local_ip_for_outbound_sockets=127.18.113.6
--webserver_interface=127.18.113.6
--webserver_port=0
--tserver_master_addrs=127.18.113.62:38409
--builtin_ntp_servers=127.18.113.20:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 01:55:24.592895 18884 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu as pid 19637
I20240502 01:55:24.593101 18884 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:55:24.612483 18884 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 01:55:24.778177 18884 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 01:55:24.781280 18884 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6de331880e14463c8794f5044fcd96f2 to finish bootstrapping
I20240502 01:55:24.792809 18884 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ff5e229761fa4416a4b4fcf88066f209 to finish bootstrapping
I20240502 01:55:24.802278 18884 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7e0ec58bc2e2461f90c5b03c3bd0a508 to finish bootstrapping
I20240502 01:55:24.813042 18884 ts_itest-base.cc:233] Waiting for 0 tablets on tserver decc769ca2ff4be68bea9efa42a8fb69 to finish bootstrapping
I20240502 01:55:24.823894 18884 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 853a6868e15d4473910f3fe760ee4305 to finish bootstrapping
I20240502 01:55:24.832832 18884 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1ec07d14611b4de182cafd0ffe09999b to finish bootstrapping
I20240502 01:55:24.844125 18884 test_util.cc:268] Using random seed: -444228565
W20240502 01:55:52.524955 20849 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:55:49.521485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.113.6:44635 (local address 127.0.0.1:47834)
0502 01:55:49.521674 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0502 01:55:49.521697 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0502 01:55:49.522606 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:55:52.523717 (+3001111us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.18.113.6:44635: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":42}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 8 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 0 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 4 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | HEALTHY | <none> | 7 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 1 | 0
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.6:44635: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.6:44635: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.4 | experimental | 127.18.113.4:35571
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.113.4:35571
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
Tablet 90fbb67c95e24ffe936469b3a4c919fd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571): RUNNING
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
All reported replicas are:
A = decc769ca2ff4be68bea9efa42a8fb69
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | -1 | Yes
B | A B* C | 3 | -1 | Yes
C | [config not available] | | |
Tablet 26cbe67c5ab24b878b6978381dbf1a8b of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable [LEADER]
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | 4 | Yes
B | A B C* | 2 | 4 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 19 | 0 | 1 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 19
Third Quartile | 20
Maximum | 20
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 0 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 8 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 0 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 5 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | HEALTHY | <none> | 7 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.4 | experimental | 127.18.113.4:35571
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.113.4:35571
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 19
Third Quartile | 20
Maximum | 20
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:55:57.420222 18884 raft_consensus_stress-itest.cc:258] completed iteration 0
W20240502 01:56:19.404886 22035 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:56:16.219515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.113.6:44635 (local address 127.0.0.1:38818)
0502 01:56:16.219690 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0502 01:56:16.219711 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0502 01:56:16.219843 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:56:19.404651 (+3184808us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.18.113.6:44635: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":50}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 16 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 0 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 1 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 4 | 0
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.6:44635: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.6:44635: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.4 | experimental | 127.18.113.4:35571
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.113.4:35571
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
Tablet 26cbe67c5ab24b878b6978381dbf1a8b of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
7e0ec58bc2e2461f90c5b03c3bd0a508 (127.18.113.3:38429): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 7e0ec58bc2e2461f90c5b03c3bd0a508
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 6 | 21 | Yes
B | [config not available] | | |
C | A* B C | 6 | 21 | Yes
Tablet d15eccf8095a4d7481a91fa06c206eaf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
7e0ec58bc2e2461f90c5b03c3bd0a508 (127.18.113.3:38429): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 7e0ec58bc2e2461f90c5b03c3bd0a508
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 56 | Yes
B | [config not available] | | |
C | A* B C | 1 | 56 | Yes
Tablet 10628aa2951240b28f7c78824675a7e7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 57 | Yes
B | [config not available] | | |
C | A* B C | 2 | 57 | Yes
Tablet 289b9fa1e7b54bdca587a3fc2bc9ee58 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 60 | Yes
B | [config not available] | | |
C | A* B C | 4 | 60 | Yes
Tablet c8fdad9fe3484b5eb44b6cfc43e451bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 58 | Yes
B | [config not available] | | |
C | A* B C | 3 | 58 | Yes
Tablet c9947483df514a1aa36f33d64bc4e992 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 60 | Yes
B | [config not available] | | |
C | A* B C | 1 | 60 | Yes
Tablet 8ec4ecdb0aad473982537b3c4ef63687 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 60 | Yes
B | [config not available] | | |
C | A* B C | 2 | 60 | Yes
Tablet cf59b52f07a04707a6a91b1bd6a5cfd0 of table 'RemoveReplaceInCycle' is conflicted: 2 replicas' active configs disagree with the leader master's
decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571): RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
ff5e229761fa4416a4b4fcf88066f209 (127.18.113.2:43425): RUNNING
7e0ec58bc2e2461f90c5b03c3bd0a508 (127.18.113.3:38429): RUNNING [NONVOTER]
All reported replicas are:
A = decc769ca2ff4be68bea9efa42a8fb69
B = 6de331880e14463c8794f5044fcd96f2
C = ff5e229761fa4416a4b4fcf88066f209
D = 7e0ec58bc2e2461f90c5b03c3bd0a508
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D | 4 | 63 | Yes
B | A B* C D~ | 4 | 61 | Yes
C | B* C D | 4 | 64 | Yes
D | A B* C D~ | 4 | 61 | Yes
Tablet 6bb28cbfaccd486abfba1a71530cbd32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 55 | Yes
B | [config not available] | | |
C | A* B C | 1 | 55 | Yes
Tablet 25b07f5f458c4594a54dab00c84fc7fa of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 57 | Yes
B | [config not available] | | |
C | A* B C | 2 | 57 | Yes
Tablet 33d561324df84688b90ddf30278b1603 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 59 | Yes
B | [config not available] | | |
C | A* B C | 1 | 59 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | CONSENSUS_MISMATCH | 20 | 10 | 0 | 9 | 1
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 9
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 64
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 0 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 15 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 0 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 1 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | HEALTHY | <none> | 1 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.4 | experimental | 127.18.113.4:35571
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.113.4:35571
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | 1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:56:23.799197 18884 raft_consensus_stress-itest.cc:258] completed iteration 1
W20240502 01:56:32.716073 22744 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:56:29.648443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.113.5:40911 (local address 127.0.0.1:52434)
0502 01:56:29.649524 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0502 01:56:29.649542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 01:56:29.649637 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:56:32.715723 (+3066086us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.18.113.5:40911: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":30891,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 13 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 1 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.5:40911: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.5:40911: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.4 | experimental | 127.18.113.4:35571
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.113.4:35571
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.4:35571, and 3 other server(s)
Tablet Summary
Tablet 26cbe67c5ab24b878b6978381dbf1a8b of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING
7e0ec58bc2e2461f90c5b03c3bd0a508 (127.18.113.3:38429): RUNNING
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING [LEADER]
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 7e0ec58bc2e2461f90c5b03c3bd0a508
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 7 | 27 | Yes
B | A B C* | 7 | 27 | Yes
C | A B C* | 7 | 27 | Yes
Tablet ab6f11913a6242919f38d71b1630cef9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING
ff5e229761fa4416a4b4fcf88066f209 (127.18.113.2:43425): RUNNING [LEADER]
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = ff5e229761fa4416a4b4fcf88066f209
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 60 | Yes
B | A B* C | 4 | 60 | Yes
C | [config not available] | | |
Tablet 10628aa2951240b28f7c78824675a7e7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 66 | Yes
B | [config not available] | | |
C | A* B C | 2 | 66 | Yes
Tablet 289b9fa1e7b54bdca587a3fc2bc9ee58 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 69 | Yes
B | [config not available] | | |
C | A B* C | 5 | 69 | Yes
Tablet e0ba43e115104622a573cf15b2e20ffc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
7e0ec58bc2e2461f90c5b03c3bd0a508 (127.18.113.3:38429): RUNNING
All reported replicas are:
A = 853a6868e15d4473910f3fe760ee4305
B = 6de331880e14463c8794f5044fcd96f2
C = 7e0ec58bc2e2461f90c5b03c3bd0a508
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 5 | 57 | Yes
C | A B* C | 5 | 57 | Yes
Tablet c8fdad9fe3484b5eb44b6cfc43e451bf of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 67 | Yes
B | [config not available] | | |
C | A* B C | 3 | 67 | Yes
Tablet c9947483df514a1aa36f33d64bc4e992 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 69 | Yes
B | [config not available] | | |
C | A* B C | 1 | 69 | Yes
Tablet 8ec4ecdb0aad473982537b3c4ef63687 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 68 | Yes
B | [config not available] | | |
C | A* B C | 2 | 68 | Yes
Tablet 6bb28cbfaccd486abfba1a71530cbd32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 64 | Yes
B | [config not available] | | |
C | A* B C | 1 | 64 | Yes
Tablet f7409ef1398f425a957d4a93cd7767e9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING
ff5e229761fa4416a4b4fcf88066f209 (127.18.113.2:43425): RUNNING [LEADER]
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = ff5e229761fa4416a4b4fcf88066f209
C = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 56 | Yes
B | A B* C | 4 | 56 | Yes
C | [config not available] | | |
Tablet 25b07f5f458c4594a54dab00c84fc7fa of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 67 | Yes
B | [config not available] | | |
C | A B* C | 3 | 67 | Yes
Tablet 90fbb67c95e24ffe936469b3a4c919fd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable [LEADER]
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING
7e0ec58bc2e2461f90c5b03c3bd0a508 (127.18.113.3:38429): RUNNING
All reported replicas are:
A = 853a6868e15d4473910f3fe760ee4305
B = 6de331880e14463c8794f5044fcd96f2
C = 7e0ec58bc2e2461f90c5b03c3bd0a508
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 8 | 63 | Yes
C | A* B C | 8 | 63 | Yes
Tablet 33d561324df84688b90ddf30278b1603 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): TS unavailable
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 853a6868e15d4473910f3fe760ee4305
C = 1ec07d14611b4de182cafd0ffe09999b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 68 | Yes
B | [config not available] | | |
C | A* B C | 1 | 68 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 8 | 0 | 12 | 0
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
F20240502 01:56:40.060159 23084 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-10628aa2951240b28f7c78824675a7e7.queue_time_us, tablet-download-pool-10628aa2951240b28f7c78824675a7e7.queue_time_us) (tablet-download-pool-10628aa2951240b28f7c78824675a7e7.run_cpu_time_us, tablet-download-pool-10628aa2951240b28f7c78824675a7e7.run_cpu_time_us) (tablet-download-pool-10628aa2951240b28f7c78824675a7e7.run_wall_time_us, tablet-download-pool-10628aa2951240b28f7c78824675a7e7.run_wall_time_us) (tablet-download-pool-1bf019a515d749648565e82aa1f9a0ca.queue_time_us, tablet-download-pool-1bf019a515d749648565e82aa1f9a0ca.queue_time_us) (tablet-download-pool-1bf019a515d749648565e82aa1f9a0ca.run_cpu_time_us, tablet-download-pool-1bf019a515d749648565e82aa1f9a0ca.run_cpu_time_us) (tablet-download-pool-1bf019a515d749648565e82aa1f9a0ca.run_wall_time_us, tablet-download-pool-1bf019a515d749648565e82aa1f9a0ca.run_wall_time_us) (tablet-download-pool-25b07f5f458c4594a54dab00c84fc7fa.queue_time_us, tablet-download-pool-25b07f5f458c4594a54dab00c84fc7fa.queue_time_us) (tablet-download-pool-25b07f5f458c4594a54dab00c84fc7fa.run_cpu_time_us, tablet-download-pool-25b07f5f458c4594a54dab00c84fc7fa.run_cpu_time_us) (tablet-download-pool-25b07f5f458c4594a54dab00c84fc7fa.run_wall_time_us, tablet-download-pool-25b07f5f458c4594a54dab00c84fc7fa.run_wall_time_us) (tablet-download-pool-26cbe67c5ab24b878b6978381dbf1a8b.queue_time_us, tablet-download-pool-26cbe67c5ab24b878b6978381dbf1a8b.queue_time_us) (tablet-download-pool-26cbe67c5ab24b878b6978381dbf1a8b.run_cpu_time_us, tablet-download-pool-26cbe67c5ab24b878b6978381dbf1a8b.run_cpu_time_us) (tablet-download-pool-26cbe67c5ab24b878b6978381dbf1a8b.run_wall_time_us, tablet-download-pool-26cbe67c5ab24b878b6978381dbf1a8b.run_wall_time_us) (tablet-download-pool-289b9fa1e7b54bdca587a3fc2bc9ee58.queue_time_us, tablet-download-pool-289b9fa1e7b54bdca587a3fc2bc9ee58.queue_time_us) (tablet-download-pool-289b9fa1e7b54bdca587a3fc2bc9ee58.run_cpu_time_us, tablet-download-pool-289b9fa1e7b54bdca587a3fc2bc9ee58.run_cpu_time_us) (tablet-download-pool-289b9fa1e7b54bdca587a3fc2bc9ee58.run_wall_time_us, tablet-download-pool-289b9fa1e7b54bdca587a3fc2bc9ee58.run_wall_time_us) (tablet-download-pool-33d561324df84688b90ddf30278b1603.queue_time_us, tablet-download-pool-33d561324df84688b90ddf30278b1603.queue_time_us) (tablet-download-pool-33d561324df84688b90ddf30278b1603.run_cpu_time_us, tablet-download-pool-33d561324df84688b90ddf30278b1603.run_cpu_time_us) (tablet-download-pool-33d561324df84688b90ddf30278b1603.run_wall_time_us, tablet-download-pool-33d561324df84688b90ddf30278b1603.run_wall_time_us) (tablet-download-pool-6bb28cbfaccd486abfba1a71530cbd32.queue_time_us, tablet-download-pool-6bb28cbfaccd486abfba1a71530cbd32.queue_time_us) (tablet-download-pool-6bb28cbfaccd486abfba1a71530cbd32.run_cpu_time_us, tablet-download-pool-6bb28cbfaccd486abfba1a71530cbd32.run_cpu_time_us) (tablet-download-pool-6bb28cbfaccd486abfba1a71530cbd32.run_wall_time_us, tablet-download-pool-6bb28cbfaccd486abfba1a71530cbd32.run_wall_time_us) (tablet-download-pool-86a1dad484674b29b2aa881f8fe36864.queue_time_us, tablet-download-pool-86a1dad484674b29b2aa881f8fe36864.queue_time_us) (tablet-download-pool-86a1dad484674b29b2aa881f8fe36864.run_cpu_time_us, tablet-download-pool-86a1dad484674b29b2aa881f8fe36864.run_cpu_time_us) (tablet-download-pool-86a1dad484674b29b2aa881f8fe36864.run_wall_time_us, tablet-download-pool-86a1dad484674b29b2aa881f8fe36864.run_wall_time_us) (tablet-download-pool-8ec4ecdb0aad473982537b3c4ef63687.queue_time_us, tablet-download-pool-8ec4ecdb0aad473982537b3c4ef63687.queue_time_us) (tablet-download-pool-8ec4ecdb0aad473982537b3c4ef63687.run_cpu_time_us, tablet-download-pool-8ec4ecdb0aad473982537b3c4ef63687.run_cpu_time_us) (tablet-download-pool-8ec4ecdb0aad473982537b3c4ef63687.run_wall_time_us, tablet-download-pool-8ec4ecdb0aad473982537b3c4ef63687.run_wall_time_us) (tablet-download-pool-90fbb67c95e24ffe936469b3a4c919fd.queue_time_us, tablet-download-pool-90fbb67c95e24ffe936469b3a4c919fd.queue_time_us) (tablet-download-pool-90fbb67c95e24ffe936469b3a4c919fd.run_cpu_time_us, tablet-download-pool-90fbb67c95e24ffe936469b3a4c919fd.run_cpu_time_us) (tablet-download-pool-90fbb67c95e24ffe936469b3a4c919fd.run_wall_time_us, tablet-download-pool-90fbb67c95e24ffe936469b3a4c919fd.run_wall_time_us) (tablet-download-pool-c8fdad9fe3484b5eb44b6cfc43e451bf.queue_time_us, tablet-download-pool-c8fdad9fe3484b5eb44b6cfc43e451bf.queue_time_us) (tablet-download-pool-c8fdad9fe3484b5eb44b6cfc43e451bf.run_cpu_time_us, tablet-download-pool-c8fdad9fe3484b5eb44b6cfc43e451bf.run_cpu_time_us) (tablet-download-pool-c8fdad9fe3484b5eb44b6cfc43e451bf.run_wall_time_us, tablet-download-pool-c8fdad9fe3484b5eb44b6cfc43e451bf.run_wall_time_us) (tablet-download-pool-c9947483df514a1aa36f33d64bc4e992.queue_time_us, tablet-download-pool-c9947483df514a1aa36f33d64bc4e992.queue_time_us) (tablet-download-pool-c9947483df514a1aa36f33d64bc4e992.run_cpu_time_us, tablet-download-pool-c9947483df514a1aa36f33d64bc4e992.run_cpu_time_us) (tablet-download-pool-c9947483df514a1aa36f33d64bc4e992.run_wall_time_us, tablet-download-pool-c9947483df514a1aa36f33d64bc4e992.run_wall_time_us) (tablet-download-pool-d15eccf8095a4d7481a91fa06c206eaf.queue_time_us, tablet-download-pool-d15eccf8095a4d7481a91fa06c206eaf.queue_time_us) (tablet-download-pool-d15eccf8095a4d7481a91fa06c206eaf.run_cpu_time_us, tablet-download-pool-d15eccf8095a4d7481a91fa06c206eaf.run_cpu_time_us) (tablet-download-pool-d15eccf8095a4d7481a91fa06c206eaf.run_wall_time_us, tablet-download-pool-d15eccf8095a4d7481a91fa06c206eaf.run_wall_time_us) (tablet-download-pool-e0ba43e115104622a573cf15b2e20ffc.queue_time_us, tablet-download-pool-e0ba43e115104622a573cf15b2e20ffc.queue_time_us) (tablet-download-pool-f7409ef1398f425a957d4a93cd7767e9.queue_time_us, tablet-download-pool-f7409ef1398f425a957d4a93cd7767e9.queue_time_us) (tablet-download-pool-f7409ef1398f425a957d4a93cd7767e9.run_cpu_time_us, tablet-download-pool-f7409ef1398f425a957d4a93cd7767e9.run_cpu_time_us) (tablet-download-pool-f7409ef1398f425a957d4a93cd7767e9.run_wall_time_us, tablet-download-pool-f7409ef1398f425a957d4a93cd7767e9.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f6353fb7dcd google::LogMessage::Fail() at ??:0
@ 0x7f6353fbbb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6353fb77cc google::LogMessage::Flush() at ??:0
@ 0x7f6353fb8f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6356cf8742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f6356cc4f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f6356cbecd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f63661118de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f6366129daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f6366254480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f636628ca8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f636626f981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f636626f70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f6356cc8b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6356ccc19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6356ccc019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f6356ca701e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f635a02c6db start_thread at ??:0
@ 0x7f6352e6a71f clone at ??:0
W20240502 01:56:41.863790 19799 meta_cache.cc:302] tablet ab6f11913a6242919f38d71b1630cef9: replica decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571) has failed: Network error: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
Tablet 289b9fa1e7b54bdca587a3fc2bc9ee58 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING [NONVOTER]
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = decc769ca2ff4be68bea9efa42a8fb69
D = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 6 | 76 | Yes
B | A* B C D~ | 6 | 76 | Yes
C | [config not available] | | |
D | A* B C D~ | 6 | 76 | Yes
Tablet 6bb28cbfaccd486abfba1a71530cbd32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING [NONVOTER]
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = decc769ca2ff4be68bea9efa42a8fb69
D = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 1 | 70 | Yes
B | A* B C D~ | 1 | 70 | Yes
C | [config not available] | | |
D | A* B C D~ | 1 | 70 | Yes
Tablet f7409ef1398f425a957d4a93cd7767e9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
ff5e229761fa4416a4b4fcf88066f209 (127.18.113.2:43425): RUNNING
decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571): TS unavailable
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = ff5e229761fa4416a4b4fcf88066f209
C = decc769ca2ff4be68bea9efa42a8fb69
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 6 | 62 | Yes
B | A* B C | 6 | 62 | Yes
C | [config not available] | | |
Tablet 25b07f5f458c4594a54dab00c84fc7fa of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
1ec07d14611b4de182cafd0ffe09999b (127.18.113.6:44635): RUNNING
decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING [NONVOTER]
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = 1ec07d14611b4de182cafd0ffe09999b
C = decc769ca2ff4be68bea9efa42a8fb69
D = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 74 | Yes
B | A* B C D~ | 4 | 74 | Yes
C | [config not available] | | |
D | A* B C D~ | 4 | 74 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 16 | 0 | 4 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 9
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 66
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.2:43425, and 3 other server(s)
Tablet Summary
Tablet f7409ef1398f425a957d4a93cd7767e9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
ff5e229761fa4416a4b4fcf88066f209 (127.18.113.2:43425): RUNNING
decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571): TS unavailable
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = ff5e229761fa4416a4b4fcf88066f209
C = decc769ca2ff4be68bea9efa42a8fb69
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 6 | 62 | Yes
B | A* B C | 6 | 62 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 19 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 9
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
Tablet f7409ef1398f425a957d4a93cd7767e9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
6de331880e14463c8794f5044fcd96f2 (127.18.113.1:40473): RUNNING [LEADER]
ff5e229761fa4416a4b4fcf88066f209 (127.18.113.2:43425): RUNNING
decc769ca2ff4be68bea9efa42a8fb69 (127.18.113.4:35571): TS unavailable
853a6868e15d4473910f3fe760ee4305 (127.18.113.5:40911): RUNNING [NONVOTER]
All reported replicas are:
A = 6de331880e14463c8794f5044fcd96f2
B = ff5e229761fa4416a4b4fcf88066f209
C = decc769ca2ff4be68bea9efa42a8fb69
D = 853a6868e15d4473910f3fe760ee4305
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 6 | 69 | Yes
B | A* B C D~ | 6 | 69 | Yes
C | [config not available] | | |
D | A* B C D~ | 6 | 69 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 19 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 64
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.3:38429, tserver@127.18.113.6:44635, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.3:38429, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.2:43425, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.6:44635, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.6:44635, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.6:44635, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.3:38429, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.5:40911, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.1:40473, tserver@127.18.113.6:44635, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ef73f5956f4f4113b4bbe6a5e3859a6f | 127.18.113.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.113.20:40673 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
1ec07d14611b4de182cafd0ffe09999b | 127.18.113.6:44635 | HEALTHY | <none> | 1 | 0
6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | HEALTHY | <none> | 17 | 0
7e0ec58bc2e2461f90c5b03c3bd0a508 | 127.18.113.3:38429 | HEALTHY | <none> | 3 | 0
853a6868e15d4473910f3fe760ee4305 | 127.18.113.5:40911 | HEALTHY | <none> | 0 | 0
ff5e229761fa4416a4b4fcf88066f209 | 127.18.113.2:43425 | HEALTHY | <none> | 0 | 0
decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.18.113.4:35571: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.18.113.4:35571: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
local_ip_for_outbound_sockets | 127.18.113.1 | experimental | 127.18.113.1:40473
local_ip_for_outbound_sockets | 127.18.113.2 | experimental | 127.18.113.2:43425
local_ip_for_outbound_sockets | 127.18.113.3 | experimental | 127.18.113.3:38429
local_ip_for_outbound_sockets | 127.18.113.5 | experimental | 127.18.113.5:40911
local_ip_for_outbound_sockets | 127.18.113.6 | experimental | 127.18.113.6:44635
never_fsync | true | unsafe,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.113.1:40473
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.113.2:43425
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.113.3:38429
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.113.5:40911
server_dump_info_path | /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.113.6:44635
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.18.113.20:40673 | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
time_source | builtin | 127.18.113.6:44635, 127.18.113.1:40473, 127.18.113.3:38429, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.18.113.62:38409, tserver@127.18.113.6:44635, tserver@127.18.113.1:40473, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 10
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Big | 6de331880e14463c8794f5044fcd96f2 | 127.18.113.1:40473 | 21
Small | decc769ca2ff4be68bea9efa42a8fb69 | 127.18.113.4:35571 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
E20240502 01:57:27.452811 18884 external_mini_cluster.cc:943] Process with UUID decc769ca2ff4be68bea9efa42a8fb69 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 01:57:27.561344 18884 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu pid 18986
I20240502 01:57:28.009173 18884 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu with pid 18986
I20240502 01:57:28.078979 18884 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu pid 19117
I20240502 01:57:28.495812 18884 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu with pid 19117
I20240502 01:57:28.553927 18884 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu pid 19247
I20240502 01:57:28.928174 18884 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu with pid 19247
I20240502 01:57:28.992115 18884 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu pid 19507
I20240502 01:57:29.341717 18884 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu with pid 19507
I20240502 01:57:29.402351 18884 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu pid 19637
I20240502 01:57:29.724813 18884 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu with pid 19637
I20240502 01:57:29.778545 18884 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu pid 18896
I20240502 01:57:30.074478 18884 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksN0TNC/build/asan/bin/kudu with pid 18896
2024-05-02T01:57:30Z chronyd exiting
I20240502 01:57:30.141978 18884 test_util.cc:175] -----------------------------------------------
I20240502 01:57:30.142149 18884 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksN0TNC/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714614921028186-18884-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (129034 ms)
[----------] 1 test from RaftConsensusStressITest (129034 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (129034 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240502 01:57:30.144755 18884 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 10 messages since previous log ~48 seconds ago