[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:15:22.007239 3225 test_util.cc:268] Using random seed: 1003261594
I20240502 14:15:22.018952 3225 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:15:22.019035 3225 ts_itest-base.cc:113] --------------
I20240502 14:15:22.019081 3225 ts_itest-base.cc:114] 6 tablet servers
I20240502 14:15:22.019129 3225 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:15:22.019176 3225 ts_itest-base.cc:116] --------------
2024-05-02T14:15:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:15:22Z Disabled control of system clock
I20240502 14:15:22.045035 3225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
/tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.38.126:43745
--webserver_interface=127.3.38.126
--webserver_port=0
--builtin_ntp_servers=127.3.38.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.38.126:43745
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=20 with env {}
I20240502 14:15:22.585165 3225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu as pid 3238
I20240502 14:15:22.585399 3225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:15:22.709988 3225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
/tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.38.65:0
--local_ip_for_outbound_sockets=127.3.38.65
--webserver_interface=127.3.38.65
--webserver_port=0
--tserver_master_addrs=127.3.38.126:43745
--builtin_ntp_servers=127.3.38.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 14:15:23.178727 3225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu as pid 3329
I20240502 14:15:23.178969 3225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:15:23.183384 3225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
/tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.38.66:0
--local_ip_for_outbound_sockets=127.3.38.66
--webserver_interface=127.3.38.66
--webserver_port=0
--tserver_master_addrs=127.3.38.126:43745
--builtin_ntp_servers=127.3.38.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 14:15:23.642038 3225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu as pid 3461
I20240502 14:15:23.642218 3225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:15:23.646086 3225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
/tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.38.67:0
--local_ip_for_outbound_sockets=127.3.38.67
--webserver_interface=127.3.38.67
--webserver_port=0
--tserver_master_addrs=127.3.38.126:43745
--builtin_ntp_servers=127.3.38.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 14:15:24.095345 3225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu as pid 3592
I20240502 14:15:24.095549 3225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:15:24.099318 3225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
/tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.38.68:0
--local_ip_for_outbound_sockets=127.3.38.68
--webserver_interface=127.3.38.68
--webserver_port=0
--tserver_master_addrs=127.3.38.126:43745
--builtin_ntp_servers=127.3.38.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 14:15:24.548405 3225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu as pid 3723
I20240502 14:15:24.548610 3225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:15:24.552310 3225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
/tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.38.69:0
--local_ip_for_outbound_sockets=127.3.38.69
--webserver_interface=127.3.38.69
--webserver_port=0
--tserver_master_addrs=127.3.38.126:43745
--builtin_ntp_servers=127.3.38.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 14:15:25.012892 3225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu as pid 3854
I20240502 14:15:25.013144 3225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:15:25.017654 3225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
/tmp/dist-test-taskk_ORIX/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.38.70:0
--local_ip_for_outbound_sockets=127.3.38.70
--webserver_interface=127.3.38.70
--webserver_port=0
--tserver_master_addrs=127.3.38.126:43745
--builtin_ntp_servers=127.3.38.84:42327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 14:15:25.478885 3225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu as pid 3984
I20240502 14:15:25.479101 3225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:15:25.495891 3225 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 14:15:25.781448 3225 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 14:15:25.784073 3225 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 15eec034bc5b4c30aabdf0b047ffb596 to finish bootstrapping
I20240502 14:15:25.793517 3225 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 68eb7b43a9a74c34bc4f40d6acd3605d to finish bootstrapping
I20240502 14:15:25.802987 3225 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 597036c4b45c474cae04be9c2e042281 to finish bootstrapping
I20240502 14:15:25.809973 3225 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f873901087ca4146a925c21108f9e086 to finish bootstrapping
I20240502 14:15:25.817106 3225 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bddd1a818e934945b0f1ed33a37fb130 to finish bootstrapping
I20240502 14:15:25.825760 3225 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 66dc1bd0e3624b0689b837f309c18c9e to finish bootstrapping
I20240502 14:15:25.834450 3225 test_util.cc:268] Using random seed: 1007088804
W20240502 14:15:46.955399 4898 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:43.715219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.38.69:41631 (local address 127.0.0.1:40630)
0502 14:15:43.721448 (+ 6229us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:43.721474 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0502 14:15:43.721579 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:15:46.954147 (+3232568us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.3.38.69:41631: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":6119}
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | HEALTHY | <none> | 0 | 0
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 7 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 4 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 3 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.69:41631: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.69:41631: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.3.38.65 | experimental | 127.3.38.65:35959
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.38.65:35959
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.65:35959, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
Tablet d13b8edcb1a64ea4b357f4df1013cf13 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
All reported replicas are:
A = 66dc1bd0e3624b0689b837f309c18c9e
B = bddd1a818e934945b0f1ed33a37fb130
C = 597036c4b45c474cae04be9c2e042281
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 47 | Yes
B | [config not available] | | |
C | A B* C | 1 | 47 | Yes
Tablet fb4d8b3a9c854de9a30136082b0970b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 48 | Yes
B | A B* C | 3 | 48 | Yes
C | [config not available] | | |
Tablet 8f0458754432459293eb36a4cbfaac35 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet f45a11ff35584397b9babe225f3e92db of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet fd5eb886bba5425b92c9d67f3599f6d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 49 | Yes
B | A* B C | 1 | 49 | Yes
C | [config not available] | | |
Tablet 1a3e1bf1f6c84b4583afa2f12b2ac3b4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 08ee367cdaf44fca9e4a69ee48d52a72 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet bc785ded1566464cb71d62c6d9d1d28c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 49 | Yes
B | A* B C | 1 | 49 | Yes
C | [config not available] | | |
Tablet c2e038912a86421aaf0581015e058fa2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet b446a89204ac4245a0c4e19073ca9581 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 46 | Yes
B | A* B C | 1 | 46 | Yes
C | [config not available] | | |
Tablet c4821753f07442df9ee318d581831334 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 60a300fdbf3246a38121e2bf53835125 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 45 | Yes
B | A B* C | 2 | 45 | Yes
C | [config not available] | | |
Tablet 2c1bc838e5b34b3781c9ca28e818db5f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 5b1eb8df18a44f94b753d400cc7033c8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 4 | 47 | Yes
B | A B C | 4 | 47 | Yes
C | [config not available] | | |
Tablet 1a5b64eb6f624a86b6eb5bbc1c02a05c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet f6afb6ffb8884e2d91a496a3f12dadc3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 49 | Yes
B | A* B C | 3 | 49 | Yes
C | [config not available] | | |
Tablet e80682151afd4611bf84ee69df216be9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 2 | -1 | Yes
B | [config not available] | | |
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 3 | 0 | 17 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 4
Median | 12
Third Quartile | 17
Maximum | 20
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | HEALTHY | <none> | 6 | 0
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 9 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 0 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 0 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.3.38.65 | experimental | 127.3.38.65:35959
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.38.65:35959
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 8
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:15:57.933367 3225 raft_consensus_stress-itest.cc:258] completed iteration 0
W20240502 14:16:01.546316 5718 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:58.507143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.38.66:40403 (local address 127.0.0.1:47782)
0502 14:15:58.507277 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:58.507295 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 14:15:58.507378 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:16:01.546050 (+3038672us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.3.38.66:40403: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":33}
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | HEALTHY | <none> | 5 | 0
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 9 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 1 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.66:40403: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.66:40403: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.3.38.65 | experimental | 127.3.38.65:35959
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.38.65:35959
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.65:35959, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
Tablet b1f7855a111f48a99d65415d1362ab57 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 597036c4b45c474cae04be9c2e042281
C = 68eb7b43a9a74c34bc4f40d6acd3605d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 4 | Yes
B | A B* C | 1 | 4 | Yes
C | [config not available] | | |
Tablet 9b53949d944849728d104005213e3423 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = 68eb7b43a9a74c34bc4f40d6acd3605d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 44 | Yes
B | A* B C | 1 | 44 | Yes
C | [config not available] | | |
Tablet 8f0458754432459293eb36a4cbfaac35 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): RUNNING [LEADER]
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 48 | Yes
B | A B* C | 2 | 48 | Yes
C | [config not available] | | |
Tablet 4eeb44c658634bc0af04e33ef3819ae8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
All reported replicas are:
A = 68eb7b43a9a74c34bc4f40d6acd3605d
B = 597036c4b45c474cae04be9c2e042281
C = f873901087ca4146a925c21108f9e086
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 4 | 45 | Yes
C | A B C* | 4 | 45 | Yes
Tablet f45a11ff35584397b9babe225f3e92db of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): RUNNING [LEADER]
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 50 | Yes
B | A B* C | 2 | 50 | Yes
C | [config not available] | | |
Tablet e59254b8cb8f4402a66e0d595f5cc6bd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
All reported replicas are:
A = 68eb7b43a9a74c34bc4f40d6acd3605d
B = 597036c4b45c474cae04be9c2e042281
C = f873901087ca4146a925c21108f9e086
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 45 | Yes
C | A B* C | 3 | 45 | Yes
Tablet 08ee367cdaf44fca9e4a69ee48d52a72 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): RUNNING
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 47 | Yes
B | A* B C | 3 | 47 | Yes
C | [config not available] | | |
Tablet 2c1bc838e5b34b3781c9ca28e818db5f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): RUNNING [LEADER]
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 49 | Yes
B | A B* C | 3 | 49 | Yes
C | [config not available] | | |
Tablet 1a5b64eb6f624a86b6eb5bbc1c02a05c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): RUNNING [LEADER]
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): TS unavailable
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 48 | Yes
B | A B* C | 3 | 48 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 12 | 0 | 8 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 8
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | HEALTHY | <none> | 5 | 0
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 9 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 1 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 0 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.3.38.65 | experimental | 127.3.38.65:35959
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.38.65:35959
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 4
Median | 11
Third Quartile | 17
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:16:05.053268 3225 raft_consensus_stress-itest.cc:258] completed iteration 1
W20240502 14:16:08.714429 6171 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:16:05.530592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.38.69:41631 (local address 127.0.0.1:51002)
0502 14:16:05.530886 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0502 14:16:05.530916 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0502 14:16:05.531037 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:16:08.714086 (+3183049us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.3.38.69:41631: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":157}
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | HEALTHY | <none> | 5 | 0
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 9 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 1 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.69:41631: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.69:41631: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.3.38.65 | experimental | 127.3.38.65:35959
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.38.65:35959
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.65:35959, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
Tablet 9b53949d944849728d104005213e3423 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 51 | Yes
B | A* B C | 1 | 51 | Yes
C | [config not available] | | |
Tablet fb4d8b3a9c854de9a30136082b0970b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 56 | Yes
B | A B* C | 3 | 56 | Yes
C | [config not available] | | |
Tablet 4eeb44c658634bc0af04e33ef3819ae8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 54 | Yes
B | A B* C | 4 | 54 | Yes
C | [config not available] | | |
Tablet fd5eb886bba5425b92c9d67f3599f6d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 56 | Yes
B | A* B C | 1 | 56 | Yes
C | [config not available] | | |
Tablet 1a3e1bf1f6c84b4583afa2f12b2ac3b4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): RUNNING
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 15eec034bc5b4c30aabdf0b047ffb596
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 41 | Yes
B | [config not available] | | |
C | A* B C | 4 | 41 | Yes
Tablet e59254b8cb8f4402a66e0d595f5cc6bd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 51 | Yes
B | A* B C | 3 | 51 | Yes
C | [config not available] | | |
Tablet bc785ded1566464cb71d62c6d9d1d28c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 56 | Yes
B | A* B C | 1 | 56 | Yes
C | [config not available] | | |
Tablet b446a89204ac4245a0c4e19073ca9581 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 53 | Yes
B | A* B C | 1 | 53 | Yes
C | [config not available] | | |
Tablet 60a300fdbf3246a38121e2bf53835125 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 53 | Yes
B | A B* C | 2 | 53 | Yes
C | [config not available] | | |
Tablet 5b1eb8df18a44f94b753d400cc7033c8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 5 | 54 | Yes
B | A* B C | 5 | 54 | Yes
C | [config not available] | | |
Tablet f6afb6ffb8884e2d91a496a3f12dadc3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = f873901087ca4146a925c21108f9e086
C = bddd1a818e934945b0f1ed33a37fb130
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 57 | Yes
B | A* B C | 3 | 57 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 9 | 0 | 11 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 4
Median | 11
Third Quartile | 17
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | HEALTHY | <none> | 5 | 0
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 9 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 0 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 0 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 7 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.3.38.65 | experimental | 127.3.38.65:35959
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.38.65:35959
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 5
Median | 10
Third Quartile | 17
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:16:18.395870 3225 raft_consensus_stress-itest.cc:258] completed iteration 2
F20240502 14:16:28.259795 7188 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-08ee367cdaf44fca9e4a69ee48d52a72.queue_time_us, tablet-download-pool-08ee367cdaf44fca9e4a69ee48d52a72.queue_time_us) (tablet-download-pool-08ee367cdaf44fca9e4a69ee48d52a72.run_cpu_time_us, tablet-download-pool-08ee367cdaf44fca9e4a69ee48d52a72.run_cpu_time_us) (tablet-download-pool-08ee367cdaf44fca9e4a69ee48d52a72.run_wall_time_us, tablet-download-pool-08ee367cdaf44fca9e4a69ee48d52a72.run_wall_time_us) (tablet-download-pool-1a3e1bf1f6c84b4583afa2f12b2ac3b4.queue_time_us, tablet-download-pool-1a3e1bf1f6c84b4583afa2f12b2ac3b4.queue_time_us) (tablet-download-pool-1a3e1bf1f6c84b4583afa2f12b2ac3b4.run_cpu_time_us, tablet-download-pool-1a3e1bf1f6c84b4583afa2f12b2ac3b4.run_cpu_time_us) (tablet-download-pool-1a3e1bf1f6c84b4583afa2f12b2ac3b4.run_wall_time_us, tablet-download-pool-1a3e1bf1f6c84b4583afa2f12b2ac3b4.run_wall_time_us) (tablet-download-pool-1a5b64eb6f624a86b6eb5bbc1c02a05c.queue_time_us, tablet-download-pool-1a5b64eb6f624a86b6eb5bbc1c02a05c.queue_time_us) (tablet-download-pool-1a5b64eb6f624a86b6eb5bbc1c02a05c.run_cpu_time_us, tablet-download-pool-1a5b64eb6f624a86b6eb5bbc1c02a05c.run_cpu_time_us) (tablet-download-pool-1a5b64eb6f624a86b6eb5bbc1c02a05c.run_wall_time_us, tablet-download-pool-1a5b64eb6f624a86b6eb5bbc1c02a05c.run_wall_time_us) (tablet-download-pool-2c1bc838e5b34b3781c9ca28e818db5f.queue_time_us, tablet-download-pool-2c1bc838e5b34b3781c9ca28e818db5f.queue_time_us) (tablet-download-pool-2c1bc838e5b34b3781c9ca28e818db5f.run_cpu_time_us, tablet-download-pool-2c1bc838e5b34b3781c9ca28e818db5f.run_cpu_time_us) (tablet-download-pool-2c1bc838e5b34b3781c9ca28e818db5f.run_wall_time_us, tablet-download-pool-2c1bc838e5b34b3781c9ca28e818db5f.run_wall_time_us) (tablet-download-pool-60a300fdbf3246a38121e2bf53835125.queue_time_us, tablet-download-pool-60a300fdbf3246a38121e2bf53835125.queue_time_us) (tablet-download-pool-8f0458754432459293eb36a4cbfaac35.queue_time_us, tablet-download-pool-8f0458754432459293eb36a4cbfaac35.queue_time_us) (tablet-download-pool-8f0458754432459293eb36a4cbfaac35.run_cpu_time_us, tablet-download-pool-8f0458754432459293eb36a4cbfaac35.run_cpu_time_us) (tablet-download-pool-8f0458754432459293eb36a4cbfaac35.run_wall_time_us, tablet-download-pool-8f0458754432459293eb36a4cbfaac35.run_wall_time_us) (tablet-download-pool-b446a89204ac4245a0c4e19073ca9581.queue_time_us, tablet-download-pool-b446a89204ac4245a0c4e19073ca9581.queue_time_us) (tablet-download-pool-b446a89204ac4245a0c4e19073ca9581.run_cpu_time_us, tablet-download-pool-b446a89204ac4245a0c4e19073ca9581.run_cpu_time_us) (tablet-download-pool-b446a89204ac4245a0c4e19073ca9581.run_wall_time_us, tablet-download-pool-b446a89204ac4245a0c4e19073ca9581.run_wall_time_us) (tablet-download-pool-c2e038912a86421aaf0581015e058fa2.queue_time_us, tablet-download-pool-c2e038912a86421aaf0581015e058fa2.queue_time_us) (tablet-download-pool-c2e038912a86421aaf0581015e058fa2.run_cpu_time_us, tablet-download-pool-c2e038912a86421aaf0581015e058fa2.run_cpu_time_us) (tablet-download-pool-c2e038912a86421aaf0581015e058fa2.run_wall_time_us, tablet-download-pool-c2e038912a86421aaf0581015e058fa2.run_wall_time_us) (tablet-download-pool-c4821753f07442df9ee318d581831334.queue_time_us, tablet-download-pool-c4821753f07442df9ee318d581831334.queue_time_us) (tablet-download-pool-c4821753f07442df9ee318d581831334.run_cpu_time_us, tablet-download-pool-c4821753f07442df9ee318d581831334.run_cpu_time_us) (tablet-download-pool-c4821753f07442df9ee318d581831334.run_wall_time_us, tablet-download-pool-c4821753f07442df9ee318d581831334.run_wall_time_us) (tablet-download-pool-d13b8edcb1a64ea4b357f4df1013cf13.queue_time_us, tablet-download-pool-d13b8edcb1a64ea4b357f4df1013cf13.queue_time_us) (tablet-download-pool-d13b8edcb1a64ea4b357f4df1013cf13.run_cpu_time_us, tablet-download-pool-d13b8edcb1a64ea4b357f4df1013cf13.run_cpu_time_us) (tablet-download-pool-d13b8edcb1a64ea4b357f4df1013cf13.run_wall_time_us, tablet-download-pool-d13b8edcb1a64ea4b357f4df1013cf13.run_wall_time_us) (tablet-download-pool-e59254b8cb8f4402a66e0d595f5cc6bd.queue_time_us, tablet-download-pool-e59254b8cb8f4402a66e0d595f5cc6bd.queue_time_us) (tablet-download-pool-e59254b8cb8f4402a66e0d595f5cc6bd.run_cpu_time_us, tablet-download-pool-e59254b8cb8f4402a66e0d595f5cc6bd.run_cpu_time_us) (tablet-download-pool-e59254b8cb8f4402a66e0d595f5cc6bd.run_wall_time_us, tablet-download-pool-e59254b8cb8f4402a66e0d595f5cc6bd.run_wall_time_us) (tablet-download-pool-e80682151afd4611bf84ee69df216be9.queue_time_us, tablet-download-pool-e80682151afd4611bf84ee69df216be9.queue_time_us) (tablet-download-pool-e80682151afd4611bf84ee69df216be9.run_cpu_time_us, tablet-download-pool-e80682151afd4611bf84ee69df216be9.run_cpu_time_us) (tablet-download-pool-e80682151afd4611bf84ee69df216be9.run_wall_time_us, tablet-download-pool-e80682151afd4611bf84ee69df216be9.run_wall_time_us) (tablet-download-pool-f45a11ff35584397b9babe225f3e92db.queue_time_us, tablet-download-pool-f45a11ff35584397b9babe225f3e92db.queue_time_us) (tablet-download-pool-f45a11ff35584397b9babe225f3e92db.run_cpu_time_us, tablet-download-pool-f45a11ff35584397b9babe225f3e92db.run_cpu_time_us) (tablet-download-pool-f45a11ff35584397b9babe225f3e92db.run_wall_time_us, tablet-download-pool-f45a11ff35584397b9babe225f3e92db.run_wall_time_us) (tablet-download-pool-f6afb6ffb8884e2d91a496a3f12dadc3.queue_time_us, tablet-download-pool-f6afb6ffb8884e2d91a496a3f12dadc3.queue_time_us) (tablet-download-pool-f6afb6ffb8884e2d91a496a3f12dadc3.run_cpu_time_us, tablet-download-pool-f6afb6ffb8884e2d91a496a3f12dadc3.run_cpu_time_us) (tablet-download-pool-f6afb6ffb8884e2d91a496a3f12dadc3.run_wall_time_us, tablet-download-pool-f6afb6ffb8884e2d91a496a3f12dadc3.run_wall_time_us) (tablet-download-pool-fd5eb886bba5425b92c9d67f3599f6d9.queue_time_us, tablet-download-pool-fd5eb886bba5425b92c9d67f3599f6d9.queue_time_us) (tablet-download-pool-fd5eb886bba5425b92c9d67f3599f6d9.run_cpu_time_us, tablet-download-pool-fd5eb886bba5425b92c9d67f3599f6d9.run_cpu_time_us) (tablet-download-pool-fd5eb886bba5425b92c9d67f3599f6d9.run_wall_time_us, tablet-download-pool-fd5eb886bba5425b92c9d67f3599f6d9.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fe00b4b9dcd google::LogMessage::Fail() at ??:0
@ 0x7fe00b4bdb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe00b4b97cc google::LogMessage::Flush() at ??:0
@ 0x7fe00b4baf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe00e1fa742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fe00e1c6f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fe00e1c0cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fe01d6138de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fe01d62bdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fe01d756480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fe01d78ea8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe01d771981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fe01d77170e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe00e1cab75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe00e1ce19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe00e1ce019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fe00e1a901e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe01152e6db start_thread at ??:0
@ 0x7fe00a36c71f clone at ??:0
W20240502 14:16:29.932924 4149 meta_cache.cc:302] tablet 1a5b64eb6f624a86b6eb5bbc1c02a05c: replica 15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959) has failed: Network error: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111)
W20240502 14:16:40.458284 7639 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:16:37.312011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.38.69:41631 (local address 127.0.0.1:53012)
0502 14:16:37.312438 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0502 14:16:37.312471 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0502 14:16:37.312560 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:16:40.458057 (+3145497us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.3.38.69:41631: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":277}
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 5 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 5 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 7 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.3.38.69:41631: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.69:41631: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 2 other server(s)
Tablet Summary
Tablet b1f7855a111f48a99d65415d1362ab57 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): RUNNING
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
All reported replicas are:
A = bddd1a818e934945b0f1ed33a37fb130
B = 68eb7b43a9a74c34bc4f40d6acd3605d
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 4 | 21 | Yes
C | A B C* | 4 | 21 | Yes
Tablet d13b8edcb1a64ea4b357f4df1013cf13 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): not running [NONVOTER]
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tablet Copy: Tombstoned tablet d13b8edcb1a64ea4b357f4df1013cf13: Tablet copy aborted
All reported replicas are:
A = 15eec034bc5b4c30aabdf0b047ffb596
B = f873901087ca4146a925c21108f9e086
C = 68eb7b43a9a74c34bc4f40d6acd3605d
D = 597036c4b45c474cae04be9c2e042281
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 69 | Yes
C | A B* C D~ | 3 | 69 | Yes
D | [config not available] | | |
Tablet 9b53949d944849728d104005213e3423 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = 68eb7b43a9a74c34bc4f40d6acd3605d
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 70 | Yes
B | [config not available] | | |
C | A* B C | 3 | 70 | Yes
Tablet fb4d8b3a9c854de9a30136082b0970b8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 5 | 79 | Yes
B | [config not available] | | |
C | A* B C | 5 | 79 | Yes
Tablet 4eeb44c658634bc0af04e33ef3819ae8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable [NONVOTER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
D = 15eec034bc5b4c30aabdf0b047ffb596
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 5 | 70 | Yes
B | [config not available] | | |
C | A B* C D~ | 5 | 72 | Yes
D | [config not available] | | |
Tablet f45a11ff35584397b9babe225f3e92db of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [NONVOTER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
D = 597036c4b45c474cae04be9c2e042281
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 69 | Yes
D | A B C D~ | 3 | 69 | Yes
Tablet fd5eb886bba5425b92c9d67f3599f6d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 78 | Yes
B | [config not available] | | |
C | A B* C | 2 | 80 | Yes
Tablet 1a3e1bf1f6c84b4583afa2f12b2ac3b4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [NONVOTER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
D = 597036c4b45c474cae04be9c2e042281
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 56 | Yes
B | [config not available] | | |
C | A* B C D~ | 4 | 56 | Yes
D | A B C D~ | 4 | 56 | Yes
Tablet e59254b8cb8f4402a66e0d595f5cc6bd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 6 | 71 | Yes
B | [config not available] | | |
C | A B C* | 6 | 71 | Yes
Tablet 08ee367cdaf44fca9e4a69ee48d52a72 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [NONVOTER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
D = 597036c4b45c474cae04be9c2e042281
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 3 | 64 | Yes
B | [config not available] | | |
C | A* B C D~ | 3 | 64 | Yes
D | A B C D~ | 3 | 64 | Yes
Tablet bc785ded1566464cb71d62c6d9d1d28c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
All reported replicas are:
A = 597036c4b45c474cae04be9c2e042281
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | 77 | Yes
B | [config not available] | | |
C | A B C* | 4 | 77 | Yes
Tablet c2e038912a86421aaf0581015e058fa2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [NONVOTER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
D = 597036c4b45c474cae04be9c2e042281
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 62 | Yes
B | [config not available] | | |
C | A* B C D~ | 4 | 62 | Yes
D | A B C D~ | 4 | 62 | Yes
Tablet b446a89204ac4245a0c4e19073ca9581 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable [NONVOTER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
D = 15eec034bc5b4c30aabdf0b047ffb596
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* D~ | | | Yes
A | A B C* D~ | 3 | 77 | Yes
B | [config not available] | | |
C | A B C* D~ | 3 | 77 | Yes
D | [config not available] | | |
Tablet 60a300fdbf3246a38121e2bf53835125 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING [LEADER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | 77 | Yes
B | [config not available] | | |
C | A B C* | 3 | 77 | Yes
Tablet 5b1eb8df18a44f94b753d400cc7033c8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): RUNNING [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
All reported replicas are:
A = bddd1a818e934945b0f1ed33a37fb130
B = 597036c4b45c474cae04be9c2e042281
C = 66dc1bd0e3624b0689b837f309c18c9e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 7 | 80 | Yes
C | A B* C | 7 | 80 | Yes
Tablet f6afb6ffb8884e2d91a496a3f12dadc3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING
bddd1a818e934945b0f1ed33a37fb130 (127.3.38.69:41631): TS unavailable [LEADER]
66dc1bd0e3624b0689b837f309c18c9e (127.3.38.70:38531): RUNNING
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable [NONVOTER]
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = bddd1a818e934945b0f1ed33a37fb130
C = 66dc1bd0e3624b0689b837f309c18c9e
D = 15eec034bc5b4c30aabdf0b047ffb596
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 4 | 80 | Yes
B | [config not available] | | |
C | A B* C | 4 | 82 | No
D | [config not available] | | |
Tablet e80682151afd4611bf84ee69df216be9 of table 'RemoveReplaceInCycle' is recovering: 1 on-going tablet copies
f873901087ca4146a925c21108f9e086 (127.3.38.68:43317): RUNNING [LEADER]
15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959): TS unavailable
68eb7b43a9a74c34bc4f40d6acd3605d (127.3.38.66:40403): RUNNING
597036c4b45c474cae04be9c2e042281 (127.3.38.67:36955): not running [NONVOTER]
State: INITIALIZED
Data state: TABLET_DATA_COPYING
Last status: Tablet Copy: Replacing tablet superblock
All reported replicas are:
A = f873901087ca4146a925c21108f9e086
B = 15eec034bc5b4c30aabdf0b047ffb596
C = 68eb7b43a9a74c34bc4f40d6acd3605d
D = 597036c4b45c474cae04be9c2e042281
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 63 | Yes
B | [config not available] | | |
C | A* B C D~ | 4 | 63 | Yes
D | A B C D~ | 4 | 63 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 4 | 1 | 15 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 9
First Quartile | 11
Median | 12
Third Quartile | 12
Maximum | 17
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 17
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 9
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 72
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240502 14:16:42.886339 4149 meta_cache.cc:302] tablet e80682151afd4611bf84ee69df216be9: replica 15eec034bc5b4c30aabdf0b047ffb596 (127.3.38.65:35959) has failed: Network error: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111)
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.69:41631, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.69:41631, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.69:41631, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.66:40403, tserver@127.3.38.68:43317, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.69:41631, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.66:40403, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.66:40403, tserver@127.3.38.69:41631, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.69:41631, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.68:43317, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.66:40403, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.66:40403, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.67:36955, tserver@127.3.38.66:40403, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.66:40403, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.69:41631, tserver@127.3.38.70:38531, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.66:40403, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.68:43317, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
eede1e30e1c243639bd5955bb5a02dc0 | 127.3.38.126:43745 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.3.38.84:42327 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
597036c4b45c474cae04be9c2e042281 | 127.3.38.67:36955 | HEALTHY | <none> | 1 | 0
66dc1bd0e3624b0689b837f309c18c9e | 127.3.38.70:38531 | HEALTHY | <none> | 8 | 0
68eb7b43a9a74c34bc4f40d6acd3605d | 127.3.38.66:40403 | HEALTHY | <none> | 6 | 0
bddd1a818e934945b0f1ed33a37fb130 | 127.3.38.69:41631 | HEALTHY | <none> | 0 | 0
f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | HEALTHY | <none> | 6 | 0
15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.3.38.65:35959: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.3.38.65:35959: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
local_ip_for_outbound_sockets | 127.3.38.66 | experimental | 127.3.38.66:40403
local_ip_for_outbound_sockets | 127.3.38.67 | experimental | 127.3.38.67:36955
local_ip_for_outbound_sockets | 127.3.38.68 | experimental | 127.3.38.68:43317
local_ip_for_outbound_sockets | 127.3.38.69 | experimental | 127.3.38.69:41631
local_ip_for_outbound_sockets | 127.3.38.70 | experimental | 127.3.38.70:38531
never_fsync | true | unsafe,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.38.66:40403
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.38.67:36955
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.38.68:43317
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.38.69:41631
server_dump_info_path | /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.3.38.70:38531
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+--------------------------------------------------------------------------------
builtin_ntp_servers | 127.3.38.84:42327 | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
time_source | builtin | 127.3.38.67:36955, 127.3.38.70:38531, 127.3.38.66:40403, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.3.38.126:43745, tserver@127.3.38.70:38531, tserver@127.3.38.67:36955, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+-------------------+---------------
Big | f873901087ca4146a925c21108f9e086 | 127.3.38.68:43317 | 19
Small | 15eec034bc5b4c30aabdf0b047ffb596 | 127.3.38.65:35959 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
E20240502 14:17:23.901142 3225 external_mini_cluster.cc:943] Process with UUID 15eec034bc5b4c30aabdf0b047ffb596 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 14:17:25.377032 3225 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu pid 3461
I20240502 14:17:25.720145 3225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu with pid 3461
I20240502 14:17:25.794786 3225 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu pid 3592
I20240502 14:17:26.157966 3225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu with pid 3592
I20240502 14:17:26.213095 3225 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu pid 3723
I20240502 14:17:26.547014 3225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu with pid 3723
I20240502 14:17:26.605623 3225 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu pid 3854
I20240502 14:17:26.946564 3225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu with pid 3854
I20240502 14:17:27.002378 3225 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu pid 3984
I20240502 14:17:27.339349 3225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu with pid 3984
I20240502 14:17:27.387369 3225 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu pid 3238
I20240502 14:17:27.670840 3225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk_ORIX/build/asan/bin/kudu with pid 3238
2024-05-02T14:17:27Z chronyd exiting
I20240502 14:17:27.737948 3225 test_util.cc:175] -----------------------------------------------
I20240502 14:17:27.738029 3225 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskk_ORIX/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714659321934432-3225-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (125735 ms)
[----------] 1 test from RaftConsensusStressITest (125735 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (125735 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST