[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:25:22.907285 28797 test_util.cc:268] Using random seed: 1034357865
I20240502 21:25:22.920183 28797 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:22.920325 28797 ts_itest-base.cc:113] --------------
I20240502 21:25:22.920475 28797 ts_itest-base.cc:114] 6 tablet servers
I20240502 21:25:22.920594 28797 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:22.920734 28797 ts_itest-base.cc:116] --------------
2024-05-02T21:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:22Z Disabled control of system clock
I20240502 21:25:22.947271 28797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpuTibz/build/asan/bin/kudu
/tmp/dist-test-taskpuTibz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.31.126:36561
--webserver_interface=127.28.31.126
--webserver_port=0
--builtin_ntp_servers=127.28.31.84:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.31.126:36561
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=20 with env {}
I20240502 21:25:23.405407 28797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpuTibz/build/asan/bin/kudu as pid 28810
I20240502 21:25:23.405669 28797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:23.495344 28797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpuTibz/build/asan/bin/kudu
/tmp/dist-test-taskpuTibz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.31.65:0
--local_ip_for_outbound_sockets=127.28.31.65
--webserver_interface=127.28.31.65
--webserver_port=0
--tserver_master_addrs=127.28.31.126:36561
--builtin_ntp_servers=127.28.31.84:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 21:25:23.955013 28797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpuTibz/build/asan/bin/kudu as pid 28902
I20240502 21:25:23.955220 28797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:23.958428 28797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpuTibz/build/asan/bin/kudu
/tmp/dist-test-taskpuTibz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.31.66:0
--local_ip_for_outbound_sockets=127.28.31.66
--webserver_interface=127.28.31.66
--webserver_port=0
--tserver_master_addrs=127.28.31.126:36561
--builtin_ntp_servers=127.28.31.84:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 21:25:24.407053 28797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpuTibz/build/asan/bin/kudu as pid 29034
I20240502 21:25:24.407224 28797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:24.410334 28797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpuTibz/build/asan/bin/kudu
/tmp/dist-test-taskpuTibz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.31.67:0
--local_ip_for_outbound_sockets=127.28.31.67
--webserver_interface=127.28.31.67
--webserver_port=0
--tserver_master_addrs=127.28.31.126:36561
--builtin_ntp_servers=127.28.31.84:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 21:25:24.897780 28797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpuTibz/build/asan/bin/kudu as pid 29166
I20240502 21:25:24.897954 28797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:24.901315 28797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpuTibz/build/asan/bin/kudu
/tmp/dist-test-taskpuTibz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.31.68:0
--local_ip_for_outbound_sockets=127.28.31.68
--webserver_interface=127.28.31.68
--webserver_port=0
--tserver_master_addrs=127.28.31.126:36561
--builtin_ntp_servers=127.28.31.84:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 21:25:25.337872 28797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpuTibz/build/asan/bin/kudu as pid 29297
I20240502 21:25:25.338044 28797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:25.341921 28797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpuTibz/build/asan/bin/kudu
/tmp/dist-test-taskpuTibz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.31.69:0
--local_ip_for_outbound_sockets=127.28.31.69
--webserver_interface=127.28.31.69
--webserver_port=0
--tserver_master_addrs=127.28.31.126:36561
--builtin_ntp_servers=127.28.31.84:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 21:25:25.799516 28797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpuTibz/build/asan/bin/kudu as pid 29431
I20240502 21:25:25.799702 28797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:25.803354 28797 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpuTibz/build/asan/bin/kudu
/tmp/dist-test-taskpuTibz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.31.70:0
--local_ip_for_outbound_sockets=127.28.31.70
--webserver_interface=127.28.31.70
--webserver_port=0
--tserver_master_addrs=127.28.31.126:36561
--builtin_ntp_servers=127.28.31.84:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 21:25:26.249251 28797 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpuTibz/build/asan/bin/kudu as pid 29563
I20240502 21:25:26.249398 28797 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:25:26.266173 28797 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 21:25:26.574689 28797 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 21:25:26.577348 28797 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9fa91cd3585c48a6a151ecf785befd5f to finish bootstrapping
I20240502 21:25:26.586719 28797 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8435c3cc95a746ee8339ebfde100c878 to finish bootstrapping
I20240502 21:25:26.594731 28797 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4dedb7a585ac48b8aad4e0c610dbdde6 to finish bootstrapping
I20240502 21:25:26.603137 28797 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9e4eaa2eeead4cc1b97b3bf8c1f4a412 to finish bootstrapping
I20240502 21:25:26.610780 28797 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c14c4f0567e84cc89cf472e489443a12 to finish bootstrapping
I20240502 21:25:26.619174 28797 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 83dcaf13fb764b479c3f564b667a2982 to finish bootstrapping
I20240502 21:25:26.626101 28797 test_util.cc:268] Using random seed: 1038076681
W20240502 21:25:47.717839 30498 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:25:44.540677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.31.69:42797 (local address 127.0.0.1:36244)
0502 21:25:44.541209 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0502 21:25:44.541225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0502 21:25:44.541355 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:25:47.716466 (+3175111us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.28.31.69:42797: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":410}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 0 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | HEALTHY | <none> | 6 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 4 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.69:42797: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.69:42797: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.68 | experimental | 127.28.31.68:42915
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.28.31.68:42915
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
Tablet 5f3fa189b99a4d23b9f7060d47feeae7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING [LEADER]
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable
8435c3cc95a746ee8339ebfde100c878 (127.28.31.66:46231): RUNNING
All reported replicas are:
A = 9fa91cd3585c48a6a151ecf785befd5f
B = c14c4f0567e84cc89cf472e489443a12
C = 8435c3cc95a746ee8339ebfde100c878
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 45 | Yes
B | [config not available] | | |
C | A* B C | 2 | 45 | Yes
Tablet 495f58977f7940269bad84143c39530f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING [LEADER]
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 44 | Yes
B | [config not available] | | |
C | A* B C | 1 | 44 | Yes
Tablet 1a3a5d9af79a437890016efb80ee8541 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING [LEADER]
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 42 | Yes
B | [config not available] | | |
C | A* B C | 1 | 42 | Yes
Tablet 784ecfd76bef4473b3dd19eff795fe6f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable [LEADER]
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 43 | Yes
B | [config not available] | | |
C | A B* C | 1 | 43 | Yes
Tablet 75c986ff87dd47ada35e69ded7031870 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable [LEADER]
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 42 | Yes
B | [config not available] | | |
C | A B* C | 1 | 42 | Yes
Tablet 586e7b3a0e6e42afa786abb8d0893558 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable [LEADER]
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 42 | Yes
B | [config not available] | | |
C | A B* C | 3 | 42 | Yes
Tablet 5e68bf50b18a4fc996a1045260810164 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable [LEADER]
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 45 | Yes
B | [config not available] | | |
C | A B* C | 2 | 45 | Yes
Tablet 79f1c3356d964ba0b4353697f3688f61 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING [LEADER]
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 41 | Yes
B | [config not available] | | |
C | A* B C | 1 | 41 | Yes
Tablet 37fbf578ffca40eb96529826bc7cc0d6 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable [LEADER]
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 43 | Yes
B | [config not available] | | |
C | A B* C | 2 | 43 | Yes
Tablet bb8841469760409fa77fcfa07c6f4349 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): RUNNING [LEADER]
c14c4f0567e84cc89cf472e489443a12 (127.28.31.69:42797): TS unavailable
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = c14c4f0567e84cc89cf472e489443a12
C = 9fa91cd3585c48a6a151ecf785befd5f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 41 | Yes
B | [config not available] | | |
C | A* B C | 1 | 41 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 10 | 0 | 10 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 12
Third Quartile | 20
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 0 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 7 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | HEALTHY | <none> | 6 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 8 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.68 | experimental | 127.28.31.68:42915
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.28.31.68:42915
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 21
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:25:52.395608 28797 raft_consensus_stress-itest.cc:258] completed iteration 0
F20240502 21:26:02.294523 31304 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-00f63d012f4b40279a650239626c47d1.queue_time_us, tablet-download-pool-00f63d012f4b40279a650239626c47d1.queue_time_us) (tablet-download-pool-00f63d012f4b40279a650239626c47d1.run_cpu_time_us, tablet-download-pool-00f63d012f4b40279a650239626c47d1.run_cpu_time_us) (tablet-download-pool-00f63d012f4b40279a650239626c47d1.run_wall_time_us, tablet-download-pool-00f63d012f4b40279a650239626c47d1.run_wall_time_us) (tablet-download-pool-01ae185a99ad40c7ac92535cac4ee77c.queue_time_us, tablet-download-pool-01ae185a99ad40c7ac92535cac4ee77c.queue_time_us) (tablet-download-pool-01ae185a99ad40c7ac92535cac4ee77c.run_cpu_time_us, tablet-download-pool-01ae185a99ad40c7ac92535cac4ee77c.run_cpu_time_us) (tablet-download-pool-01ae185a99ad40c7ac92535cac4ee77c.run_wall_time_us, tablet-download-pool-01ae185a99ad40c7ac92535cac4ee77c.run_wall_time_us) (tablet-download-pool-1a3a5d9af79a437890016efb80ee8541.queue_time_us, tablet-download-pool-1a3a5d9af79a437890016efb80ee8541.queue_time_us) (tablet-download-pool-1a3a5d9af79a437890016efb80ee8541.run_cpu_time_us, tablet-download-pool-1a3a5d9af79a437890016efb80ee8541.run_cpu_time_us) (tablet-download-pool-1a3a5d9af79a437890016efb80ee8541.run_wall_time_us, tablet-download-pool-1a3a5d9af79a437890016efb80ee8541.run_wall_time_us) (tablet-download-pool-22c0bd9d117b4da481868e12eba4c746.queue_time_us, tablet-download-pool-22c0bd9d117b4da481868e12eba4c746.queue_time_us) (tablet-download-pool-22c0bd9d117b4da481868e12eba4c746.run_cpu_time_us, tablet-download-pool-22c0bd9d117b4da481868e12eba4c746.run_cpu_time_us) (tablet-download-pool-22c0bd9d117b4da481868e12eba4c746.run_wall_time_us, tablet-download-pool-22c0bd9d117b4da481868e12eba4c746.run_wall_time_us) (tablet-download-pool-2d1534cfa5f54cf7981d32a1612ff084.queue_time_us, tablet-download-pool-2d1534cfa5f54cf7981d32a1612ff084.queue_time_us) (tablet-download-pool-2d1534cfa5f54cf7981d32a1612ff084.run_cpu_time_us, tablet-download-pool-2d1534cfa5f54cf7981d32a1612ff084.run_cpu_time_us) (tablet-download-pool-2d1534cfa5f54cf7981d32a1612ff084.run_wall_time_us, tablet-download-pool-2d1534cfa5f54cf7981d32a1612ff084.run_wall_time_us) (tablet-download-pool-37fbf578ffca40eb96529826bc7cc0d6.queue_time_us, tablet-download-pool-37fbf578ffca40eb96529826bc7cc0d6.queue_time_us) (tablet-download-pool-37fbf578ffca40eb96529826bc7cc0d6.run_cpu_time_us, tablet-download-pool-37fbf578ffca40eb96529826bc7cc0d6.run_cpu_time_us) (tablet-download-pool-37fbf578ffca40eb96529826bc7cc0d6.run_wall_time_us, tablet-download-pool-37fbf578ffca40eb96529826bc7cc0d6.run_wall_time_us) (tablet-download-pool-401fc10520d04043b121d3caeca405d9.queue_time_us, tablet-download-pool-401fc10520d04043b121d3caeca405d9.queue_time_us) (tablet-download-pool-401fc10520d04043b121d3caeca405d9.run_cpu_time_us, tablet-download-pool-401fc10520d04043b121d3caeca405d9.run_cpu_time_us) (tablet-download-pool-401fc10520d04043b121d3caeca405d9.run_wall_time_us, tablet-download-pool-401fc10520d04043b121d3caeca405d9.run_wall_time_us) (tablet-download-pool-5e68bf50b18a4fc996a1045260810164.queue_time_us, tablet-download-pool-5e68bf50b18a4fc996a1045260810164.queue_time_us) (tablet-download-pool-5e68bf50b18a4fc996a1045260810164.run_cpu_time_us, tablet-download-pool-5e68bf50b18a4fc996a1045260810164.run_cpu_time_us) (tablet-download-pool-5e68bf50b18a4fc996a1045260810164.run_wall_time_us, tablet-download-pool-5e68bf50b18a4fc996a1045260810164.run_wall_time_us) (tablet-download-pool-5f3fa189b99a4d23b9f7060d47feeae7.queue_time_us, tablet-download-pool-5f3fa189b99a4d23b9f7060d47feeae7.queue_time_us) (tablet-download-pool-5f3fa189b99a4d23b9f7060d47feeae7.run_cpu_time_us, tablet-download-pool-5f3fa189b99a4d23b9f7060d47feeae7.run_cpu_time_us) (tablet-download-pool-5f3fa189b99a4d23b9f7060d47feeae7.run_wall_time_us, tablet-download-pool-5f3fa189b99a4d23b9f7060d47feeae7.run_wall_time_us) (tablet-download-pool-784ecfd76bef4473b3dd19eff795fe6f.queue_time_us, tablet-download-pool-784ecfd76bef4473b3dd19eff795fe6f.queue_time_us) (tablet-download-pool-79f1c3356d964ba0b4353697f3688f61.queue_time_us, tablet-download-pool-79f1c3356d964ba0b4353697f3688f61.queue_time_us) (tablet-download-pool-79f1c3356d964ba0b4353697f3688f61.run_cpu_time_us, tablet-download-pool-79f1c3356d964ba0b4353697f3688f61.run_cpu_time_us) (tablet-download-pool-79f1c3356d964ba0b4353697f3688f61.run_wall_time_us, tablet-download-pool-79f1c3356d964ba0b4353697f3688f61.run_wall_time_us) (tablet-download-pool-840ef33c6b68405f96d79557641b3bdc.queue_time_us, tablet-download-pool-840ef33c6b68405f96d79557641b3bdc.queue_time_us) (tablet-download-pool-840ef33c6b68405f96d79557641b3bdc.run_cpu_time_us, tablet-download-pool-840ef33c6b68405f96d79557641b3bdc.run_cpu_time_us) (tablet-download-pool-840ef33c6b68405f96d79557641b3bdc.run_wall_time_us, tablet-download-pool-840ef33c6b68405f96d79557641b3bdc.run_wall_time_us) (tablet-download-pool-c77880bba558496097dbb59fd7c5e8fb.queue_time_us, tablet-download-pool-c77880bba558496097dbb59fd7c5e8fb.queue_time_us) (tablet-download-pool-c77880bba558496097dbb59fd7c5e8fb.run_cpu_time_us, tablet-download-pool-c77880bba558496097dbb59fd7c5e8fb.run_cpu_time_us) (tablet-download-pool-c77880bba558496097dbb59fd7c5e8fb.run_wall_time_us, tablet-download-pool-c77880bba558496097dbb59fd7c5e8fb.run_wall_time_us) (tablet-download-pool-d5b04c61cf6d47b2b0d44ae987884ac7.queue_time_us, tablet-download-pool-d5b04c61cf6d47b2b0d44ae987884ac7.queue_time_us) (tablet-download-pool-d5b04c61cf6d47b2b0d44ae987884ac7.run_cpu_time_us, tablet-download-pool-d5b04c61cf6d47b2b0d44ae987884ac7.run_cpu_time_us) (tablet-download-pool-d5b04c61cf6d47b2b0d44ae987884ac7.run_wall_time_us, tablet-download-pool-d5b04c61cf6d47b2b0d44ae987884ac7.run_wall_time_us) (tablet-download-pool-e84e226e4e4a4ffcb2627cda042d9408.queue_time_us, tablet-download-pool-e84e226e4e4a4ffcb2627cda042d9408.queue_time_us) (tablet-download-pool-e84e226e4e4a4ffcb2627cda042d9408.run_cpu_time_us, tablet-download-pool-e84e226e4e4a4ffcb2627cda042d9408.run_cpu_time_us) (tablet-download-pool-e84e226e4e4a4ffcb2627cda042d9408.run_wall_time_us, tablet-download-pool-e84e226e4e4a4ffcb2627cda042d9408.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f4547d01dcd google::LogMessage::Fail() at ??:0
@ 0x7f4547d05b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4547d017cc google::LogMessage::Flush() at ??:0
@ 0x7f4547d02f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f454aa42742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f454aa0ef8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f454aa08cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f4559e5b8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f4559e73daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f4559f9e480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f4559fd6a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f4559fb9981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f4559fb970e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f454aa12b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f454aa1619c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f454aa16019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f454a9f101e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f454dd766db start_thread at ??:0
@ 0x7f4546bb471f clone at ??:0
W20240502 21:26:07.940857 31509 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:26:04.759656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.31.70:44661 (local address 127.0.0.1:39828)
0502 21:26:04.764271 (+ 4615us) negotiation.cc:107] Waiting for socket to connect
0502 21:26:04.764289 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 21:26:04.764387 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 21:26:07.940429 (+3176042us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.28.31.70:44661: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":4497,"thread_start_us":4340,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 1 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 13 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | UNAVAILABLE | <none> | n/a | n/a
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.70:44661: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.70:44661: received 0 of 4 requested bytes (UNAVAILABLE)
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.69:42797, and 2 other server(s)
Tablet Summary
Tablet 5f3fa189b99a4d23b9f7060d47feeae7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING [LEADER]
8435c3cc95a746ee8339ebfde100c878 (127.28.31.66:46231): RUNNING
83dcaf13fb764b479c3f564b667a2982 (127.28.31.70:44661): TS unavailable
All reported replicas are:
A = 9fa91cd3585c48a6a151ecf785befd5f
B = 8435c3cc95a746ee8339ebfde100c878
C = 83dcaf13fb764b479c3f564b667a2982
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 57 | Yes
B | A* B C | 2 | 57 | Yes
C | [config not available] | | |
Tablet 495f58977f7940269bad84143c39530f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915): TS unavailable [LEADER]
9fa91cd3585c48a6a151ecf785befd5f (127.28.31.65:32871): RUNNING
8435c3cc95a746ee8339ebfde100c878 (127.28.31.66:46231): RUNNING
4dedb7a585ac48b8aad4e0c610dbdde6 (127.28.31.67:35485): missing [NONVOTER]
All reported replicas are:
A = 9e4eaa2eeead4cc1b97b3bf8c1f4a412
B = 9fa91cd3585c48a6a151ecf785befd5f
C = 8435c3cc95a746ee8339ebfde100c878
D = 4dedb7a585ac48b8aad4e0c610dbdde6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | [config not available] | | |
B | A* B C D~ | 4 | 52 | Yes
C | A B C | 4 | 47 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 18 | 0 | 2 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 64
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
W20240502 21:26:10.646008 29730 meta_cache.cc:302] tablet 79f1c3356d964ba0b4353697f3688f61: replica 9e4eaa2eeead4cc1b97b3bf8c1f4a412 (127.28.31.68:42915) has failed: Network error: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.66:46231, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.66:46231, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.66:46231, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.66:46231, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.66:46231, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.65:32871, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.66:46231, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.66:46231, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.70:44661, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.70:44661, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.66:46231, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.66:46231, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.67:35485, tserver@127.28.31.66:46231, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4b6e54989f3d4303bf20c339d3f53947 | 127.28.31.126:36561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.28.31.84:38343 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+-------------+----------+----------------+-----------------
4dedb7a585ac48b8aad4e0c610dbdde6 | 127.28.31.67:35485 | HEALTHY | <none> | 1 | 0
83dcaf13fb764b479c3f564b667a2982 | 127.28.31.70:44661 | HEALTHY | <none> | 0 | 0
8435c3cc95a746ee8339ebfde100c878 | 127.28.31.66:46231 | HEALTHY | <none> | 2 | 0
9fa91cd3585c48a6a151ecf785befd5f | 127.28.31.65:32871 | HEALTHY | <none> | 12 | 0
c14c4f0567e84cc89cf472e489443a12 | 127.28.31.69:42797 | HEALTHY | <none> | 6 | 0
9e4eaa2eeead4cc1b97b3bf8c1f4a412 | 127.28.31.68:42915 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.28.31.68:42915: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.28.31.68:42915: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
local_ip_for_outbound_sockets | 127.28.31.65 | experimental | 127.28.31.65:32871
local_ip_for_outbound_sockets | 127.28.31.66 | experimental | 127.28.31.66:46231
local_ip_for_outbound_sockets | 127.28.31.67 | experimental | 127.28.31.67:35485
local_ip_for_outbound_sockets | 127.28.31.69 | experimental | 127.28.31.69:42797
local_ip_for_outbound_sockets | 127.28.31.70 | experimental | 127.28.31.70:44661
never_fsync | true | unsafe,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.31.65:32871
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.31.66:46231
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.31.67:35485
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.31.69:42797
server_dump_info_path | /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.31.70:44661
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-----------------------------------------------------------------------------------
builtin_ntp_servers | 127.28.31.84:38343 | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
time_source | builtin | 127.28.31.67:35485, 127.28.31.70:44661, 127.28.31.66:46231, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.28.31.126:36561, tserver@127.28.31.66:46231, tserver@127.28.31.67:35485, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
E20240502 21:26:49.654147 28797 external_mini_cluster.cc:943] Process with UUID 9e4eaa2eeead4cc1b97b3bf8c1f4a412 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 21:26:50.443639 28797 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpuTibz/build/asan/bin/kudu pid 28902
I20240502 21:26:50.789402 28797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpuTibz/build/asan/bin/kudu with pid 28902
I20240502 21:26:50.848026 28797 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpuTibz/build/asan/bin/kudu pid 29034
I20240502 21:26:51.205664 28797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpuTibz/build/asan/bin/kudu with pid 29034
I20240502 21:26:51.257838 28797 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpuTibz/build/asan/bin/kudu pid 29166
I20240502 21:26:51.584252 28797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpuTibz/build/asan/bin/kudu with pid 29166
I20240502 21:26:51.631101 28797 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpuTibz/build/asan/bin/kudu pid 29431
I20240502 21:26:51.933799 28797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpuTibz/build/asan/bin/kudu with pid 29431
I20240502 21:26:51.980185 28797 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpuTibz/build/asan/bin/kudu pid 29563
I20240502 21:26:52.243248 28797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpuTibz/build/asan/bin/kudu with pid 29563
I20240502 21:26:52.283267 28797 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpuTibz/build/asan/bin/kudu pid 28810
I20240502 21:26:52.555548 28797 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpuTibz/build/asan/bin/kudu with pid 28810
2024-05-02T21:26:52Z chronyd exiting
I20240502 21:26:52.614552 28797 test_util.cc:175] -----------------------------------------------
I20240502 21:26:52.614742 28797 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpuTibz/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714685122818271-28797-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (89712 ms)
[----------] 1 test from RaftConsensusStressITest (89712 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (89712 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240502 21:26:52.615949 28797 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~41 seconds ago