[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:12:46.857007 30433 test_util.cc:268] Using random seed: 222293755
I20240501 08:12:46.869992 30433 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:12:46.870091 30433 ts_itest-base.cc:113] --------------
I20240501 08:12:46.870149 30433 ts_itest-base.cc:114] 6 tablet servers
I20240501 08:12:46.870213 30433 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:12:46.870275 30433 ts_itest-base.cc:116] --------------
2024-05-01T08:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:12:46Z Disabled control of system clock
I20240501 08:12:46.895839 30433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
/tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.184.126:39727
--webserver_interface=127.29.184.126
--webserver_port=0
--builtin_ntp_servers=127.29.184.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.184.126:39727
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=20 with env {}
I20240501 08:12:47.343850 30433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu as pid 30445
I20240501 08:12:47.344131 30433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:12:47.447270 30433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
/tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.184.65:0
--local_ip_for_outbound_sockets=127.29.184.65
--webserver_interface=127.29.184.65
--webserver_port=0
--tserver_master_addrs=127.29.184.126:39727
--builtin_ntp_servers=127.29.184.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 08:12:47.925570 30433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu as pid 30535
I20240501 08:12:47.925827 30433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:12:47.930233 30433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
/tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.184.66:0
--local_ip_for_outbound_sockets=127.29.184.66
--webserver_interface=127.29.184.66
--webserver_port=0
--tserver_master_addrs=127.29.184.126:39727
--builtin_ntp_servers=127.29.184.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 08:12:48.421494 30433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu as pid 30666
I20240501 08:12:48.421680 30433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:12:48.425429 30433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
/tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.184.67:0
--local_ip_for_outbound_sockets=127.29.184.67
--webserver_interface=127.29.184.67
--webserver_port=0
--tserver_master_addrs=127.29.184.126:39727
--builtin_ntp_servers=127.29.184.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 08:12:48.894170 30433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu as pid 30796
I20240501 08:12:48.894380 30433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:12:48.897747 30433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
/tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.184.68:0
--local_ip_for_outbound_sockets=127.29.184.68
--webserver_interface=127.29.184.68
--webserver_port=0
--tserver_master_addrs=127.29.184.126:39727
--builtin_ntp_servers=127.29.184.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 08:12:49.356071 30433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu as pid 30926
I20240501 08:12:49.356256 30433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:12:49.359522 30433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
/tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.184.69:0
--local_ip_for_outbound_sockets=127.29.184.69
--webserver_interface=127.29.184.69
--webserver_port=0
--tserver_master_addrs=127.29.184.126:39727
--builtin_ntp_servers=127.29.184.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 08:12:49.797482 30433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu as pid 31056
I20240501 08:12:49.797695 30433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:12:49.800712 30433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
/tmp/dist-test-taskuFoQiG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.184.70:0
--local_ip_for_outbound_sockets=127.29.184.70
--webserver_interface=127.29.184.70
--webserver_port=0
--tserver_master_addrs=127.29.184.126:39727
--builtin_ntp_servers=127.29.184.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 08:12:50.269615 30433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu as pid 31186
I20240501 08:12:50.269766 30433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:12:50.286849 30433 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 08:12:50.504149 30433 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 08:12:50.507632 30433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 96f33b33801347528874f333281825c7 to finish bootstrapping
I20240501 08:12:50.518361 30433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ec285d285eb845c6a52b7a39d55209f2 to finish bootstrapping
I20240501 08:12:50.526350 30433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 791445d55ccc47f4885e9c8e10287fbf to finish bootstrapping
I20240501 08:12:50.535269 30433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 19b95bd157c84f2ba3ecca9d996b13a7 to finish bootstrapping
I20240501 08:12:50.544298 30433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b31fa37460024a9bb0d4e4d23fccd148 to finish bootstrapping
I20240501 08:12:50.551733 30433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8cb3fead2e2e4ac3ab580f3862992f36 to finish bootstrapping
I20240501 08:12:50.561408 30433 test_util.cc:268] Using random seed: 225998167
W20240501 08:13:11.602581 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:13:08.380514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.184.65:35771 (local address 127.0.0.1:43014)
0501 08:13:08.397244 (+ 16730us) negotiation.cc:107] Waiting for socket to connect
0501 08:13:08.397265 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 08:13:08.397510 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:13:11.601268 (+3203758us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.29.184.65:35771: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":16610,"thread_start_us":28553,"threads_started":1}
W20240501 08:13:14.673561 32122 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:13:11.603410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.184.65:35771 (local address 127.0.0.1:43028)
0501 08:13:11.603604 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0501 08:13:11.603626 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0501 08:13:11.603725 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:13:14.673246 (+3069521us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.29.184.65:35771: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":61}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | HEALTHY | <none> | 1 | 0
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 1 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 0 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 6 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 2 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.65:35771: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.65:35771: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.68 | experimental | 127.29.184.68:45791
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.184.68:45791
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.68:45791, 127.29.184.67:36849, 127.29.184.70:38603, and 2 other server(s)
time_source | builtin | 127.29.184.68:45791, 127.29.184.67:36849, 127.29.184.70:38603, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.68:45791, and 3 other server(s)
Tablet Summary
Tablet d7406d74e0ab40089b7a129b18ada957 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING [LEADER]
All reported replicas are:
A = ec285d285eb845c6a52b7a39d55209f2
B = 96f33b33801347528874f333281825c7
C = b31fa37460024a9bb0d4e4d23fccd148
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
Tablet 19c3471a67fc4c3cba8e9facea687bfb of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING
All reported replicas are:
A = 791445d55ccc47f4885e9c8e10287fbf
B = 96f33b33801347528874f333281825c7
C = b31fa37460024a9bb0d4e4d23fccd148
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 51 | Yes
B | [config not available] | | |
C | A B* C | 4 | 51 | Yes
Tablet 95363543b7a444a6a0c5c56d11ba47d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = b31fa37460024a9bb0d4e4d23fccd148
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 3 | -1 | Yes
C | A* B C | 3 | -1 | Yes
Tablet 1aea210766cd4a5aa0714a9ac564ed34 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING [LEADER]
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 14495d49c53047629b91aedfdbfeb855 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = b31fa37460024a9bb0d4e4d23fccd148
B = 96f33b33801347528874f333281825c7
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 49 | Yes
B | [config not available] | | |
C | A B* C | 2 | 49 | Yes
Tablet f30a540577194183b71e68413ac449b4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
Tablet 3fb5c33d9e5a4524a9c05bd76f82e402 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING [LEADER]
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = b31fa37460024a9bb0d4e4d23fccd148
B = 96f33b33801347528874f333281825c7
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 48 | Yes
B | [config not available] | | |
C | A* B C | 1 | 48 | Yes
Tablet 342d8ff854b34ea2b29c7d33bf491a2a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
Tablet b342e2d0513240f7bac5c683ea4d90c5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING [LEADER]
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = b31fa37460024a9bb0d4e4d23fccd148
B = 96f33b33801347528874f333281825c7
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 47 | Yes
B | [config not available] | | |
C | A* B C | 2 | 47 | Yes
Tablet 0150ebc8b7b444bcb1c1ad1ca21b23cd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
Tablet c2b09a21c6ea45878998514211b3ab19 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING [LEADER]
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = b31fa37460024a9bb0d4e4d23fccd148
B = 96f33b33801347528874f333281825c7
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 49 | Yes
B | [config not available] | | |
C | A* B C | 1 | 49 | Yes
Tablet 6dd227fe008c4a0281098c0e23f91d3e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING [LEADER]
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | A B* C | 1 | -1 | Yes
Tablet 27b9d596d27741c3876d6fbbe7ffc39d of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING [LEADER]
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = b31fa37460024a9bb0d4e4d23fccd148
B = 96f33b33801347528874f333281825c7
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 49 | Yes
B | [config not available] | | |
C | A* B C | 2 | 49 | Yes
Tablet 7eeaf58053d849a98eedc2633f5f2a76 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
Tablet 96d5170415694c2fb3c3f6b1d203c1b5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = b31fa37460024a9bb0d4e4d23fccd148
B = 96f33b33801347528874f333281825c7
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 50 | Yes
B | [config not available] | | |
C | A B* C | 2 | 50 | Yes
Tablet 9812563536f9422bbf25773b3c358fb4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING [LEADER]
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
Tablet 9466f0325a524ea58496d04364e7db81 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING [LEADER]
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = b31fa37460024a9bb0d4e4d23fccd148
B = 96f33b33801347528874f333281825c7
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A B C | 2 | 46 | Yes
B | [config not available] | | |
C | A* B C | 1 | 46 | Yes
Tablet 4e87ca2766914a128578aa9272b2fd7e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
Tablet d8ae7a4cf5f3470b9c8584ed42f3a6da of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791): RUNNING [LEADER]
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
All reported replicas are:
A = 19b95bd157c84f2ba3ecca9d996b13a7
B = b31fa37460024a9bb0d4e4d23fccd148
C = 96f33b33801347528874f333281825c7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 47 | Yes
B | A* B C | 3 | 47 | Yes
C | [config not available] | | |
Tablet 9286eff6aff3478f9b39c5af61b43453 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable [LEADER]
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = 96f33b33801347528874f333281825c7
B = ec285d285eb845c6a52b7a39d55209f2
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
Tablet 3525534ec90842db9c7098d66d591719 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING [LEADER]
96f33b33801347528874f333281825c7 (127.29.184.65:35771): TS unavailable
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING
All reported replicas are:
A = b31fa37460024a9bb0d4e4d23fccd148
B = 96f33b33801347528874f333281825c7
C = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 44 | Yes
B | [config not available] | | |
C | A* B C | 2 | 44 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 0 | 0 | 20 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 12
Third Quartile | 18
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | HEALTHY | <none> | 2 | 0
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 10 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 0 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 0 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 0 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 9 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.68 | experimental | 127.29.184.68:45791
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.184.68:45791
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 12
Third Quartile | 20
Maximum | 20
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:13:22.552369 30433 raft_consensus_stress-itest.cc:258] completed iteration 0
F20240501 08:13:26.003460 482 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0150ebc8b7b444bcb1c1ad1ca21b23cd.queue_time_us, tablet-download-pool-0150ebc8b7b444bcb1c1ad1ca21b23cd.queue_time_us) (tablet-download-pool-0150ebc8b7b444bcb1c1ad1ca21b23cd.run_cpu_time_us, tablet-download-pool-0150ebc8b7b444bcb1c1ad1ca21b23cd.run_cpu_time_us) (tablet-download-pool-0150ebc8b7b444bcb1c1ad1ca21b23cd.run_wall_time_us, tablet-download-pool-0150ebc8b7b444bcb1c1ad1ca21b23cd.run_wall_time_us) (tablet-download-pool-1aea210766cd4a5aa0714a9ac564ed34.queue_time_us, tablet-download-pool-1aea210766cd4a5aa0714a9ac564ed34.queue_time_us) (tablet-download-pool-1aea210766cd4a5aa0714a9ac564ed34.run_cpu_time_us, tablet-download-pool-1aea210766cd4a5aa0714a9ac564ed34.run_cpu_time_us) (tablet-download-pool-1aea210766cd4a5aa0714a9ac564ed34.run_wall_time_us, tablet-download-pool-1aea210766cd4a5aa0714a9ac564ed34.run_wall_time_us) (tablet-download-pool-27b9d596d27741c3876d6fbbe7ffc39d.queue_time_us, tablet-download-pool-27b9d596d27741c3876d6fbbe7ffc39d.queue_time_us) (tablet-download-pool-27b9d596d27741c3876d6fbbe7ffc39d.run_cpu_time_us, tablet-download-pool-27b9d596d27741c3876d6fbbe7ffc39d.run_cpu_time_us) (tablet-download-pool-27b9d596d27741c3876d6fbbe7ffc39d.run_wall_time_us, tablet-download-pool-27b9d596d27741c3876d6fbbe7ffc39d.run_wall_time_us) (tablet-download-pool-342d8ff854b34ea2b29c7d33bf491a2a.queue_time_us, tablet-download-pool-342d8ff854b34ea2b29c7d33bf491a2a.queue_time_us) (tablet-download-pool-342d8ff854b34ea2b29c7d33bf491a2a.run_cpu_time_us, tablet-download-pool-342d8ff854b34ea2b29c7d33bf491a2a.run_cpu_time_us) (tablet-download-pool-342d8ff854b34ea2b29c7d33bf491a2a.run_wall_time_us, tablet-download-pool-342d8ff854b34ea2b29c7d33bf491a2a.run_wall_time_us) (tablet-download-pool-3525534ec90842db9c7098d66d591719.queue_time_us, tablet-download-pool-3525534ec90842db9c7098d66d591719.queue_time_us) (tablet-download-pool-3525534ec90842db9c7098d66d591719.run_cpu_time_us, tablet-download-pool-3525534ec90842db9c7098d66d591719.run_cpu_time_us) (tablet-download-pool-3525534ec90842db9c7098d66d591719.run_wall_time_us, tablet-download-pool-3525534ec90842db9c7098d66d591719.run_wall_time_us) (tablet-download-pool-4e87ca2766914a128578aa9272b2fd7e.queue_time_us, tablet-download-pool-4e87ca2766914a128578aa9272b2fd7e.queue_time_us) (tablet-download-pool-4e87ca2766914a128578aa9272b2fd7e.run_cpu_time_us, tablet-download-pool-4e87ca2766914a128578aa9272b2fd7e.run_cpu_time_us) (tablet-download-pool-4e87ca2766914a128578aa9272b2fd7e.run_wall_time_us, tablet-download-pool-4e87ca2766914a128578aa9272b2fd7e.run_wall_time_us) (tablet-download-pool-6dd227fe008c4a0281098c0e23f91d3e.queue_time_us, tablet-download-pool-6dd227fe008c4a0281098c0e23f91d3e.queue_time_us) (tablet-download-pool-6dd227fe008c4a0281098c0e23f91d3e.run_cpu_time_us, tablet-download-pool-6dd227fe008c4a0281098c0e23f91d3e.run_cpu_time_us) (tablet-download-pool-6dd227fe008c4a0281098c0e23f91d3e.run_wall_time_us, tablet-download-pool-6dd227fe008c4a0281098c0e23f91d3e.run_wall_time_us) (tablet-download-pool-7eeaf58053d849a98eedc2633f5f2a76.queue_time_us, tablet-download-pool-7eeaf58053d849a98eedc2633f5f2a76.queue_time_us) (tablet-download-pool-7eeaf58053d849a98eedc2633f5f2a76.run_cpu_time_us, tablet-download-pool-7eeaf58053d849a98eedc2633f5f2a76.run_cpu_time_us) (tablet-download-pool-7eeaf58053d849a98eedc2633f5f2a76.run_wall_time_us, tablet-download-pool-7eeaf58053d849a98eedc2633f5f2a76.run_wall_time_us) (tablet-download-pool-9286eff6aff3478f9b39c5af61b43453.queue_time_us, tablet-download-pool-9286eff6aff3478f9b39c5af61b43453.queue_time_us) (tablet-download-pool-9286eff6aff3478f9b39c5af61b43453.run_cpu_time_us, tablet-download-pool-9286eff6aff3478f9b39c5af61b43453.run_cpu_time_us) (tablet-download-pool-9286eff6aff3478f9b39c5af61b43453.run_wall_time_us, tablet-download-pool-9286eff6aff3478f9b39c5af61b43453.run_wall_time_us) (tablet-download-pool-9466f0325a524ea58496d04364e7db81.queue_time_us, tablet-download-pool-9466f0325a524ea58496d04364e7db81.queue_time_us) (tablet-download-pool-9466f0325a524ea58496d04364e7db81.run_cpu_time_us, tablet-download-pool-9466f0325a524ea58496d04364e7db81.run_cpu_time_us) (tablet-download-pool-9466f0325a524ea58496d04364e7db81.run_wall_time_us, tablet-download-pool-9466f0325a524ea58496d04364e7db81.run_wall_time_us) (tablet-download-pool-95363543b7a444a6a0c5c56d11ba47d9.queue_time_us, tablet-download-pool-95363543b7a444a6a0c5c56d11ba47d9.queue_time_us) (tablet-download-pool-95363543b7a444a6a0c5c56d11ba47d9.run_cpu_time_us, tablet-download-pool-95363543b7a444a6a0c5c56d11ba47d9.run_cpu_time_us) (tablet-download-pool-95363543b7a444a6a0c5c56d11ba47d9.run_wall_time_us, tablet-download-pool-95363543b7a444a6a0c5c56d11ba47d9.run_wall_time_us) (tablet-download-pool-9812563536f9422bbf25773b3c358fb4.queue_time_us, tablet-download-pool-9812563536f9422bbf25773b3c358fb4.queue_time_us) (tablet-download-pool-9812563536f9422bbf25773b3c358fb4.run_cpu_time_us, tablet-download-pool-9812563536f9422bbf25773b3c358fb4.run_cpu_time_us) (tablet-download-pool-9812563536f9422bbf25773b3c358fb4.run_wall_time_us, tablet-download-pool-9812563536f9422bbf25773b3c358fb4.run_wall_time_us) (tablet-download-pool-c2b09a21c6ea45878998514211b3ab19.queue_time_us, tablet-download-pool-c2b09a21c6ea45878998514211b3ab19.queue_time_us) (tablet-download-pool-d7406d74e0ab40089b7a129b18ada957.queue_time_us, tablet-download-pool-d7406d74e0ab40089b7a129b18ada957.queue_time_us) (tablet-download-pool-d7406d74e0ab40089b7a129b18ada957.run_cpu_time_us, tablet-download-pool-d7406d74e0ab40089b7a129b18ada957.run_cpu_time_us) (tablet-download-pool-d7406d74e0ab40089b7a129b18ada957.run_wall_time_us, tablet-download-pool-d7406d74e0ab40089b7a129b18ada957.run_wall_time_us) (tablet-download-pool-f30a540577194183b71e68413ac449b4.queue_time_us, tablet-download-pool-f30a540577194183b71e68413ac449b4.queue_time_us) (tablet-download-pool-f30a540577194183b71e68413ac449b4.run_cpu_time_us, tablet-download-pool-f30a540577194183b71e68413ac449b4.run_cpu_time_us) (tablet-download-pool-f30a540577194183b71e68413ac449b4.run_wall_time_us, tablet-download-pool-f30a540577194183b71e68413ac449b4.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f1ef30acdcd google::LogMessage::Fail() at ??:0
@ 0x7f1ef30b0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f1ef30ac7cc google::LogMessage::Flush() at ??:0
@ 0x7f1ef30adf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1ef5ded742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f1ef5db9f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f1ef5db3cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f1f052068de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f1f0521edaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f1f05349480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f1f05381a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f1f05364981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f1f0536470e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1ef5dbdb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1ef5dc119c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1ef5dc1019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1ef5d9c01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1ef91216db start_thread at ??:0
@ 0x7f1ef1f5f71f clone at ??:0
W20240501 08:13:33.967638 31349 meta_cache.cc:302] tablet 1aea210766cd4a5aa0714a9ac564ed34: replica 19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791) has failed: Network error: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111)
W20240501 08:13:38.225574 890 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:13:35.009129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.184.70:38603 (local address 127.0.0.1:39920)
0501 08:13:35.018865 (+ 9736us) negotiation.cc:107] Waiting for socket to connect
0501 08:13:35.018881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 08:13:35.019023 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:13:38.225235 (+3206212us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.29.184.70:38603: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":9656,"thread_start_us":31882,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 10 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 8 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 1 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 1 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.29.184.70:38603: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.70:38603: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.65:35771, and 2 other server(s)
Tablet Summary
Tablet d7406d74e0ab40089b7a129b18ada957 of table 'TestTable' is unavailable: 2 replica(s) not RUNNING
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
8cb3fead2e2e4ac3ab580f3862992f36 (127.29.184.70:38603): TS unavailable [LEADER]
19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791): TS unavailable
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): missing [NONVOTER]
All reported replicas are:
A = ec285d285eb845c6a52b7a39d55209f2
B = 8cb3fead2e2e4ac3ab580f3862992f36
C = 19b95bd157c84f2ba3ecca9d996b13a7
D = 791445d55ccc47f4885e9c8e10287fbf
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 4 | 11 | Yes
B | [config not available] | | |
C | [config not available] | | |
D | [config not available] | | |
Tablet 95363543b7a444a6a0c5c56d11ba47d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING [LEADER]
19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791): TS unavailable
8cb3fead2e2e4ac3ab580f3862992f36 (127.29.184.70:38603): TS unavailable [NONVOTER]
All reported replicas are:
A = ec285d285eb845c6a52b7a39d55209f2
B = 791445d55ccc47f4885e9c8e10287fbf
C = 19b95bd157c84f2ba3ecca9d996b13a7
D = 8cb3fead2e2e4ac3ab580f3862992f36
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 55 | Yes
B | A B* C D~ | 6 | 55 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 1aea210766cd4a5aa0714a9ac564ed34 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING [LEADER]
19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791): TS unavailable
All reported replicas are:
A = ec285d285eb845c6a52b7a39d55209f2
B = 791445d55ccc47f4885e9c8e10287fbf
C = 19b95bd157c84f2ba3ecca9d996b13a7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 44 | Yes
B | A B* C | 3 | 44 | Yes
C | [config not available] | | |
Tablet 0150ebc8b7b444bcb1c1ad1ca21b23cd of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING [LEADER]
19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791): TS unavailable
All reported replicas are:
A = ec285d285eb845c6a52b7a39d55209f2
B = 791445d55ccc47f4885e9c8e10287fbf
C = 19b95bd157c84f2ba3ecca9d996b13a7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 47 | Yes
B | A B* C | 4 | 47 | Yes
C | [config not available] | | |
Tablet 4e87ca2766914a128578aa9272b2fd7e of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING [LEADER]
19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791): TS unavailable
b31fa37460024a9bb0d4e4d23fccd148 (127.29.184.69:33047): RUNNING [NONVOTER]
All reported replicas are:
A = ec285d285eb845c6a52b7a39d55209f2
B = 791445d55ccc47f4885e9c8e10287fbf
C = 19b95bd157c84f2ba3ecca9d996b13a7
D = b31fa37460024a9bb0d4e4d23fccd148
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 3 | 48 | Yes
B | A B* C D~ | 3 | 48 | Yes
C | [config not available] | | |
D | A B* C D~ | 3 | 48 | Yes
Tablet 9286eff6aff3478f9b39c5af61b43453 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING [LEADER]
19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791): TS unavailable
All reported replicas are:
A = ec285d285eb845c6a52b7a39d55209f2
B = 791445d55ccc47f4885e9c8e10287fbf
C = 19b95bd157c84f2ba3ecca9d996b13a7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 43 | Yes
B | A B* C | 4 | 43 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 15 | 0 | 5 | 0
TestTable | 3 | UNAVAILABLE | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 6
Median | 10
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 66
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240501 08:13:40.475284 31349 meta_cache.cc:302] tablet 4e87ca2766914a128578aa9272b2fd7e: replica 19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791) has failed: Network error: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111)
W20240501 08:13:40.698787 31349 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.29.184.126:39727, user_credentials={real_user=slave}} blocked reactor thread for 53855.1us
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
Tablet 95363543b7a444a6a0c5c56d11ba47d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
ec285d285eb845c6a52b7a39d55209f2 (127.29.184.66:32879): RUNNING
791445d55ccc47f4885e9c8e10287fbf (127.29.184.67:36849): RUNNING [LEADER]
19b95bd157c84f2ba3ecca9d996b13a7 (127.29.184.68:45791): TS unavailable
8cb3fead2e2e4ac3ab580f3862992f36 (127.29.184.70:38603): RUNNING [NONVOTER]
All reported replicas are:
A = ec285d285eb845c6a52b7a39d55209f2
B = 791445d55ccc47f4885e9c8e10287fbf
C = 19b95bd157c84f2ba3ecca9d996b13a7
D = 8cb3fead2e2e4ac3ab580f3862992f36
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C D~ | 6 | 59 | Yes
B | A B* C D~ | 6 | 59 | Yes
C | [config not available] | | |
D | A B C D~ | 6 | 59 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 19 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 64
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.69:33047, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.65:35771, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.66:32879, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.69:33047, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.69:33047, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.69:33047, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.66:32879, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.66:32879, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.70:38603, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.67:36849, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.67:36849, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
fcc5c6f75b0b4a0cad35e92ef3c4a466 | 127.29.184.126:39727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.184.84:34029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
791445d55ccc47f4885e9c8e10287fbf | 127.29.184.67:36849 | HEALTHY | <none> | 8 | 0
8cb3fead2e2e4ac3ab580f3862992f36 | 127.29.184.70:38603 | HEALTHY | <none> | 1 | 0
96f33b33801347528874f333281825c7 | 127.29.184.65:35771 | HEALTHY | <none> | 9 | 0
b31fa37460024a9bb0d4e4d23fccd148 | 127.29.184.69:33047 | HEALTHY | <none> | 3 | 0
ec285d285eb845c6a52b7a39d55209f2 | 127.29.184.66:32879 | HEALTHY | <none> | 0 | 0
19b95bd157c84f2ba3ecca9d996b13a7 | 127.29.184.68:45791 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.29.184.68:45791: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.29.184.68:45791: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
local_ip_for_outbound_sockets | 127.29.184.65 | experimental | 127.29.184.65:35771
local_ip_for_outbound_sockets | 127.29.184.66 | experimental | 127.29.184.66:32879
local_ip_for_outbound_sockets | 127.29.184.67 | experimental | 127.29.184.67:36849
local_ip_for_outbound_sockets | 127.29.184.69 | experimental | 127.29.184.69:33047
local_ip_for_outbound_sockets | 127.29.184.70 | experimental | 127.29.184.70:38603
never_fsync | true | unsafe,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.184.65:35771
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.184.66:32879
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.184.67:36849
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.184.69:33047
server_dump_info_path | /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.184.70:38603
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.29.184.84:34029 | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
time_source | builtin | 127.29.184.67:36849, 127.29.184.70:38603, 127.29.184.65:35771, and 2 other server(s)
Version Summary
Version | Servers
-----------------+--------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.29.184.126:39727, tserver@127.29.184.70:38603, tserver@127.29.184.65:35771, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 11
Third Quartile | 19
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
E20240501 08:14:21.799428 30433 external_mini_cluster.cc:943] Process with UUID 19b95bd157c84f2ba3ecca9d996b13a7 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240501 08:14:22.009438 30433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu pid 30535
I20240501 08:14:22.351351 30433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu with pid 30535
I20240501 08:14:22.400981 30433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu pid 30666
I20240501 08:14:22.741365 30433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu with pid 30666
I20240501 08:14:22.797993 30433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu pid 30796
I20240501 08:14:23.123324 30433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu with pid 30796
I20240501 08:14:23.180016 30433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu pid 31056
I20240501 08:14:23.496961 30433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu with pid 31056
I20240501 08:14:23.542950 30433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu pid 31186
I20240501 08:14:23.822441 30433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu with pid 31186
I20240501 08:14:23.864629 30433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu pid 30445
I20240501 08:14:24.128007 30433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuFoQiG/build/asan/bin/kudu with pid 30445
2024-05-01T08:14:24Z chronyd exiting
I20240501 08:14:24.186758 30433 test_util.cc:175] -----------------------------------------------
I20240501 08:14:24.186838 30433 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskuFoQiG/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714551166780475-30433-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (97335 ms)
[----------] 1 test from RaftConsensusStressITest (97335 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (97335 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240501 08:14:24.187800 30433 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 1 messages since previous log ~43 seconds ago