[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:01:33.667088 2284 test_util.cc:268] Using random seed: -325732636
I20240501 14:01:33.678483 2284 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:01:33.678565 2284 ts_itest-base.cc:113] --------------
I20240501 14:01:33.678619 2284 ts_itest-base.cc:114] 6 tablet servers
I20240501 14:01:33.678683 2284 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:01:33.678754 2284 ts_itest-base.cc:116] --------------
2024-05-01T14:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:01:33Z Disabled control of system clock
I20240501 14:01:33.703369 2284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUTCidK/build/asan/bin/kudu
/tmp/dist-test-taskUTCidK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.59.62:38965
--webserver_interface=127.2.59.62
--webserver_port=0
--builtin_ntp_servers=127.2.59.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.59.62:38965
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=20 with env {}
I20240501 14:01:34.109751 2284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUTCidK/build/asan/bin/kudu as pid 2296
I20240501 14:01:34.110029 2284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:01:34.223830 2284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUTCidK/build/asan/bin/kudu
/tmp/dist-test-taskUTCidK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.59.1:0
--local_ip_for_outbound_sockets=127.2.59.1
--webserver_interface=127.2.59.1
--webserver_port=0
--tserver_master_addrs=127.2.59.62:38965
--builtin_ntp_servers=127.2.59.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 14:01:34.682026 2284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUTCidK/build/asan/bin/kudu as pid 2386
I20240501 14:01:34.682231 2284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:01:34.685678 2284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUTCidK/build/asan/bin/kudu
/tmp/dist-test-taskUTCidK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.59.2:0
--local_ip_for_outbound_sockets=127.2.59.2
--webserver_interface=127.2.59.2
--webserver_port=0
--tserver_master_addrs=127.2.59.62:38965
--builtin_ntp_servers=127.2.59.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 14:01:35.111517 2284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUTCidK/build/asan/bin/kudu as pid 2517
I20240501 14:01:35.111697 2284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:01:35.114961 2284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUTCidK/build/asan/bin/kudu
/tmp/dist-test-taskUTCidK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.59.3:0
--local_ip_for_outbound_sockets=127.2.59.3
--webserver_interface=127.2.59.3
--webserver_port=0
--tserver_master_addrs=127.2.59.62:38965
--builtin_ntp_servers=127.2.59.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 14:01:35.541874 2284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUTCidK/build/asan/bin/kudu as pid 2647
I20240501 14:01:35.542083 2284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:01:35.545385 2284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUTCidK/build/asan/bin/kudu
/tmp/dist-test-taskUTCidK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.59.4:0
--local_ip_for_outbound_sockets=127.2.59.4
--webserver_interface=127.2.59.4
--webserver_port=0
--tserver_master_addrs=127.2.59.62:38965
--builtin_ntp_servers=127.2.59.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 14:01:35.981938 2284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUTCidK/build/asan/bin/kudu as pid 2777
I20240501 14:01:35.982125 2284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:01:35.985222 2284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUTCidK/build/asan/bin/kudu
/tmp/dist-test-taskUTCidK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.59.5:0
--local_ip_for_outbound_sockets=127.2.59.5
--webserver_interface=127.2.59.5
--webserver_port=0
--tserver_master_addrs=127.2.59.62:38965
--builtin_ntp_servers=127.2.59.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 14:01:36.410998 2284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUTCidK/build/asan/bin/kudu as pid 2907
I20240501 14:01:36.411242 2284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:01:36.414570 2284 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUTCidK/build/asan/bin/kudu
/tmp/dist-test-taskUTCidK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.59.6:0
--local_ip_for_outbound_sockets=127.2.59.6
--webserver_interface=127.2.59.6
--webserver_port=0
--tserver_master_addrs=127.2.59.62:38965
--builtin_ntp_servers=127.2.59.20:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240501 14:01:36.841674 2284 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUTCidK/build/asan/bin/kudu as pid 3037
I20240501 14:01:36.841845 2284 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:01:36.857825 2284 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 14:01:37.040609 2284 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 14:01:37.043110 2284 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1fc76a588ef94962b1098ef548930a7a to finish bootstrapping
I20240501 14:01:37.052187 2284 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8e70ad2eb3794816bfe677d38ddf729e to finish bootstrapping
I20240501 14:01:37.060303 2284 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ca463f8d8c824dcbbef9d371959005a8 to finish bootstrapping
I20240501 14:01:37.067052 2284 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 87a29ac48d5c4336a84fe85849b9f9ce to finish bootstrapping
I20240501 14:01:37.074496 2284 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 71fb1ea6da59487aaab9fc97a86fe8a5 to finish bootstrapping
I20240501 14:01:37.082177 2284 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 58b12df3d325441687a7137795377032 to finish bootstrapping
I20240501 14:01:37.090415 2284 test_util.cc:268] Using random seed: -322309305
W20240501 14:01:51.519001 3731 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:01:48.314142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.59.3:44853 (local address 127.0.0.1:34018)
0501 14:01:48.314344 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0501 14:01:48.314364 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0501 14:01:48.314473 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:01:51.517716 (+3203243us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.59.3:44853: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":61}
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 3 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 7 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 3 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | HEALTHY | <none> | 0 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 3 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.3:44853: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.3:44853: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.4 | experimental | 127.2.59.4:41237
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.59.4:41237
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
Tablet 1fa3219391e9489f9a63f6721bd439d7 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING [LEADER]
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable
All reported replicas are:
A = 87a29ac48d5c4336a84fe85849b9f9ce
B = 1fc76a588ef94962b1098ef548930a7a
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 49ae66ad88f14c8cbbef641d7e220ce3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING [LEADER]
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = ca463f8d8c824dcbbef9d371959005a8
C = 8e70ad2eb3794816bfe677d38ddf729e
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 42 | Yes
B | [config not available] | | |
C | A* B C | 2 | 42 | Yes
Tablet b3855f12777d4da2a45005df14af19d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable [LEADER]
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet 84451437534844e2a12070bf4192c230 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 8a929c14ce3e4afe9a1b21410905db35 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING [LEADER]
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 834ff224416141fe94237961dae90698 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet de52e522444c4c9296c80d0dd39f84dc of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable [LEADER]
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet af80b3fd1e344c3da23057eb18e48cae of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable [LEADER]
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet 119e22dfdc5e45de98533476886dac4a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable [LEADER]
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 3 | -1 | Yes
B | A B C* | 3 | -1 | Yes
C | [config not available] | | |
Tablet bc78439215134ac3b22d3c79f3904078 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable [LEADER]
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet e4853a7585fb4cbf97550aa324804a6f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): TS unavailable
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 8e70ad2eb3794816bfe677d38ddf729e
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | -1 | Yes
B | A B* C | 3 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 10 | 0 | 10 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 10
Median | 11
Third Quartile | 11
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 0 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 8 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | HEALTHY | <none> | 4 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 7 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.4 | experimental | 127.2.59.4:41237
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.59.4:41237
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 2
Median | 12
Third Quartile | 19
Maximum | 20
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:02:02.645244 2284 raft_consensus_stress-itest.cc:258] completed iteration 0
W20240501 14:02:17.374161 4862 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:14.259985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.59.4:41237 (local address 127.0.0.1:35800)
0501 14:02:14.260178 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:14.260247 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:14.260335 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:14.282986 (+ 22651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 14:02:14.282991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 14:02:14.283017 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0501 14:02:14.283430 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 14:02:14.283441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 14:02:14.289404 (+ 5963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 14:02:14.289410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 14:02:14.303086 (+ 13676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 14:02:14.303102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 14:02:14.303219 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 14:02:14.304357 (+ 1138us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 14:02:14.304360 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 14:02:14.304378 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 14:02:14.304420 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 14:02:17.373856 (+3069436us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.59.4:41237: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":60}
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 0 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 10 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 5 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 0 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.4 | experimental | 127.2.59.4:41237
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.59.4:41237
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.5:44551, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
Tablet 1fa3219391e9489f9a63f6721bd439d7 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING [LEADER]
All reported replicas are:
A = 87a29ac48d5c4336a84fe85849b9f9ce
B = 71fb1ea6da59487aaab9fc97a86fe8a5
C = 58b12df3d325441687a7137795377032
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 5 | 9 | Yes
C | A B C* | 5 | 9 | Yes
Tablet 84451437534844e2a12070bf4192c230 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable [LEADER]
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 87a29ac48d5c4336a84fe85849b9f9ce
C = 58b12df3d325441687a7137795377032
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 50 | Yes
B | [config not available] | | |
C | A B* C | 2 | 50 | Yes
Tablet 8a929c14ce3e4afe9a1b21410905db35 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [LEADER]
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 71fb1ea6da59487aaab9fc97a86fe8a5
C = 87a29ac48d5c4336a84fe85849b9f9ce
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 57 | Yes
B | A B* C | 4 | 57 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 18 | 0 | 2 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 20
Third Quartile | 20
Maximum | 20
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 0 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 9 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 6 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | HEALTHY | <none> | 0 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 6 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.4 | experimental | 127.2.59.4:41237
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.59.4:41237
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 21
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:02:22.400947 2284 raft_consensus_stress-itest.cc:258] completed iteration 1
W20240501 14:02:32.221978 5545 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:29.072780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.59.1:44619 (local address 127.0.0.1:55080)
0501 14:02:29.072998 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:29.073019 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:29.073269 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:32.221738 (+3148469us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.59.1:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":107}
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 0 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 11 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | HEALTHY | <none> | 0 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 10 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 0 | 0
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.1:44619: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.1:44619: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.4 | experimental | 127.2.59.4:41237
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.59.4:41237
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.4:41237, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
Tablet 1fa3219391e9489f9a63f6721bd439d7 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [LEADER]
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 6 | 17 | Yes
B | A* B C | 6 | 17 | Yes
C | [config not available] | | |
Tablet 49ae66ad88f14c8cbbef641d7e220ce3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 71fb1ea6da59487aaab9fc97a86fe8a5
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 60 | Yes
B | A* B C | 3 | 60 | Yes
C | [config not available] | | |
Tablet 1fa13beb477d4e7f8cb2e23b40b9ad7f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [LEADER]
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 54 | Yes
B | A* B C | 2 | 54 | Yes
C | [config not available] | | |
Tablet ef83bf9db5aa432493376156741250c0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [LEADER]
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 5 | 60 | Yes
B | A* B C | 5 | 60 | Yes
C | [config not available] | | |
Tablet 052ee510f96544cebd7d765d6564d1d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 54 | Yes
B | A B* C | 2 | 54 | Yes
C | [config not available] | | |
Tablet 350c1957d7074c5d9ae07bccd54e589a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [LEADER]
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 57 | Yes
B | A* B C | 3 | 57 | Yes
C | [config not available] | | |
Tablet 075d3725e1e94ad99828fc6782374d21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 57 | Yes
B | A B* C | 3 | 57 | Yes
C | [config not available] | | |
Tablet ffec49d3e436496c9a2586660c496010 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [LEADER]
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 61 | Yes
B | A* B C | 2 | 61 | Yes
C | [config not available] | | |
Tablet b633bdcdedd548e89baf5a1d1907f2c3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 54 | Yes
B | A B* C | 4 | 54 | Yes
C | [config not available] | | |
Tablet 3f4c7140c09841448258f2633a5ee7db of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 53 | Yes
B | A B* C | 2 | 53 | Yes
C | [config not available] | | |
Tablet 0feb41f93a3b44c68210b751a40a01e2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [LEADER]
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 8e70ad2eb3794816bfe677d38ddf729e
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 52 | Yes
B | A* B C | 1 | 52 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 10 | 0 | 10 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 11
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 0 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 1 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 0 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | HEALTHY | <none> | 3 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 15 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.4 | experimental | 127.2.59.4:41237
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.59.4:41237
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 9
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 21
Small | 71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:02:42.717728 2284 raft_consensus_stress-itest.cc:258] completed iteration 2
F20240501 14:02:46.295493 6402 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-052ee510f96544cebd7d765d6564d1d9.queue_time_us, tablet-download-pool-052ee510f96544cebd7d765d6564d1d9.queue_time_us) (tablet-download-pool-052ee510f96544cebd7d765d6564d1d9.run_cpu_time_us, tablet-download-pool-052ee510f96544cebd7d765d6564d1d9.run_cpu_time_us) (tablet-download-pool-052ee510f96544cebd7d765d6564d1d9.run_wall_time_us, tablet-download-pool-052ee510f96544cebd7d765d6564d1d9.run_wall_time_us) (tablet-download-pool-075d3725e1e94ad99828fc6782374d21.queue_time_us, tablet-download-pool-075d3725e1e94ad99828fc6782374d21.queue_time_us) (tablet-download-pool-075d3725e1e94ad99828fc6782374d21.run_cpu_time_us, tablet-download-pool-075d3725e1e94ad99828fc6782374d21.run_cpu_time_us) (tablet-download-pool-075d3725e1e94ad99828fc6782374d21.run_wall_time_us, tablet-download-pool-075d3725e1e94ad99828fc6782374d21.run_wall_time_us) (tablet-download-pool-0feb41f93a3b44c68210b751a40a01e2.queue_time_us, tablet-download-pool-0feb41f93a3b44c68210b751a40a01e2.queue_time_us) (tablet-download-pool-0feb41f93a3b44c68210b751a40a01e2.run_cpu_time_us, tablet-download-pool-0feb41f93a3b44c68210b751a40a01e2.run_cpu_time_us) (tablet-download-pool-0feb41f93a3b44c68210b751a40a01e2.run_wall_time_us, tablet-download-pool-0feb41f93a3b44c68210b751a40a01e2.run_wall_time_us) (tablet-download-pool-119e22dfdc5e45de98533476886dac4a.queue_time_us, tablet-download-pool-119e22dfdc5e45de98533476886dac4a.queue_time_us) (tablet-download-pool-119e22dfdc5e45de98533476886dac4a.run_cpu_time_us, tablet-download-pool-119e22dfdc5e45de98533476886dac4a.run_cpu_time_us) (tablet-download-pool-119e22dfdc5e45de98533476886dac4a.run_wall_time_us, tablet-download-pool-119e22dfdc5e45de98533476886dac4a.run_wall_time_us) (tablet-download-pool-834ff224416141fe94237961dae90698.queue_time_us, tablet-download-pool-834ff224416141fe94237961dae90698.queue_time_us) (tablet-download-pool-834ff224416141fe94237961dae90698.run_cpu_time_us, tablet-download-pool-834ff224416141fe94237961dae90698.run_cpu_time_us) (tablet-download-pool-834ff224416141fe94237961dae90698.run_wall_time_us, tablet-download-pool-834ff224416141fe94237961dae90698.run_wall_time_us) (tablet-download-pool-84451437534844e2a12070bf4192c230.queue_time_us, tablet-download-pool-84451437534844e2a12070bf4192c230.queue_time_us) (tablet-download-pool-84451437534844e2a12070bf4192c230.run_cpu_time_us, tablet-download-pool-84451437534844e2a12070bf4192c230.run_cpu_time_us) (tablet-download-pool-84451437534844e2a12070bf4192c230.run_wall_time_us, tablet-download-pool-84451437534844e2a12070bf4192c230.run_wall_time_us) (tablet-download-pool-8a929c14ce3e4afe9a1b21410905db35.queue_time_us, tablet-download-pool-8a929c14ce3e4afe9a1b21410905db35.queue_time_us) (tablet-download-pool-8a929c14ce3e4afe9a1b21410905db35.run_cpu_time_us, tablet-download-pool-8a929c14ce3e4afe9a1b21410905db35.run_cpu_time_us) (tablet-download-pool-8a929c14ce3e4afe9a1b21410905db35.run_wall_time_us, tablet-download-pool-8a929c14ce3e4afe9a1b21410905db35.run_wall_time_us) (tablet-download-pool-af80b3fd1e344c3da23057eb18e48cae.queue_time_us, tablet-download-pool-af80b3fd1e344c3da23057eb18e48cae.queue_time_us) (tablet-download-pool-af80b3fd1e344c3da23057eb18e48cae.run_cpu_time_us, tablet-download-pool-af80b3fd1e344c3da23057eb18e48cae.run_cpu_time_us) (tablet-download-pool-af80b3fd1e344c3da23057eb18e48cae.run_wall_time_us, tablet-download-pool-af80b3fd1e344c3da23057eb18e48cae.run_wall_time_us) (tablet-download-pool-b3855f12777d4da2a45005df14af19d9.queue_time_us, tablet-download-pool-b3855f12777d4da2a45005df14af19d9.queue_time_us) (tablet-download-pool-b3855f12777d4da2a45005df14af19d9.run_cpu_time_us, tablet-download-pool-b3855f12777d4da2a45005df14af19d9.run_cpu_time_us) (tablet-download-pool-b3855f12777d4da2a45005df14af19d9.run_wall_time_us, tablet-download-pool-b3855f12777d4da2a45005df14af19d9.run_wall_time_us) (tablet-download-pool-b633bdcdedd548e89baf5a1d1907f2c3.queue_time_us, tablet-download-pool-b633bdcdedd548e89baf5a1d1907f2c3.queue_time_us) (tablet-download-pool-b633bdcdedd548e89baf5a1d1907f2c3.run_cpu_time_us, tablet-download-pool-b633bdcdedd548e89baf5a1d1907f2c3.run_cpu_time_us) (tablet-download-pool-b633bdcdedd548e89baf5a1d1907f2c3.run_wall_time_us, tablet-download-pool-b633bdcdedd548e89baf5a1d1907f2c3.run_wall_time_us) (tablet-download-pool-bc78439215134ac3b22d3c79f3904078.queue_time_us, tablet-download-pool-bc78439215134ac3b22d3c79f3904078.queue_time_us) (tablet-download-pool-bc78439215134ac3b22d3c79f3904078.run_cpu_time_us, tablet-download-pool-bc78439215134ac3b22d3c79f3904078.run_cpu_time_us) (tablet-download-pool-bc78439215134ac3b22d3c79f3904078.run_wall_time_us, tablet-download-pool-bc78439215134ac3b22d3c79f3904078.run_wall_time_us) (tablet-download-pool-de52e522444c4c9296c80d0dd39f84dc.queue_time_us, tablet-download-pool-de52e522444c4c9296c80d0dd39f84dc.queue_time_us) (tablet-download-pool-de52e522444c4c9296c80d0dd39f84dc.run_cpu_time_us, tablet-download-pool-de52e522444c4c9296c80d0dd39f84dc.run_cpu_time_us) (tablet-download-pool-de52e522444c4c9296c80d0dd39f84dc.run_wall_time_us, tablet-download-pool-de52e522444c4c9296c80d0dd39f84dc.run_wall_time_us) (tablet-download-pool-e4853a7585fb4cbf97550aa324804a6f.queue_time_us, tablet-download-pool-e4853a7585fb4cbf97550aa324804a6f.queue_time_us) (tablet-download-pool-e4853a7585fb4cbf97550aa324804a6f.run_cpu_time_us, tablet-download-pool-e4853a7585fb4cbf97550aa324804a6f.run_cpu_time_us) (tablet-download-pool-e4853a7585fb4cbf97550aa324804a6f.run_wall_time_us, tablet-download-pool-e4853a7585fb4cbf97550aa324804a6f.run_wall_time_us) (tablet-download-pool-ef83bf9db5aa432493376156741250c0.queue_time_us, tablet-download-pool-ef83bf9db5aa432493376156741250c0.queue_time_us) (tablet-download-pool-ffec49d3e436496c9a2586660c496010.queue_time_us, tablet-download-pool-ffec49d3e436496c9a2586660c496010.queue_time_us) (tablet-download-pool-ffec49d3e436496c9a2586660c496010.run_cpu_time_us, tablet-download-pool-ffec49d3e436496c9a2586660c496010.run_cpu_time_us) (tablet-download-pool-ffec49d3e436496c9a2586660c496010.run_wall_time_us, tablet-download-pool-ffec49d3e436496c9a2586660c496010.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f53ea5e5dcd google::LogMessage::Fail() at ??:0
@ 0x7f53ea5e9b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f53ea5e57cc google::LogMessage::Flush() at ??:0
@ 0x7f53ea5e6f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f53ed326742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f53ed2f2f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f53ed2eccd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f53fc73f8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f53fc757daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f53fc882480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f53fc8baa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f53fc89d981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f53fc89d70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f53ed2f6b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f53ed2fa19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f53ed2fa019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f53ed2d501e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f53f065a6db start_thread at ??:0
@ 0x7f53e949871f clone at ??:0
W20240501 14:02:48.035692 3201 meta_cache.cc:302] tablet 834ff224416141fe94237961dae90698: replica 87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237) has failed: Network error: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111)
W20240501 14:02:58.333881 6788 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:02:55.270253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.59.1:44619 (local address 127.0.0.1:40530)
0501 14:02:55.270424 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0501 14:02:55.270443 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0501 14:02:55.270575 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:02:58.333657 (+3063082us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.59.1:44619: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":37}
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 7 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 1 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 8 | 0
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | UNAVAILABLE | <none> | n/a | n/a
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.1:44619: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.1:44619: received 0 of 4 requested bytes (UNAVAILABLE)
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.2:34389, and 2 other server(s)
Tablet Summary
Tablet 1fa3219391e9489f9a63f6721bd439d7 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 71fb1ea6da59487aaab9fc97a86fe8a5
B = 58b12df3d325441687a7137795377032
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 9 | 27 | Yes
B | A B* C | 9 | 27 | Yes
C | [config not available] | | |
Tablet 519afd501b344a23b067e1dd087a63fe of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING
All reported replicas are:
A = 58b12df3d325441687a7137795377032
B = 1fc76a588ef94962b1098ef548930a7a
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 59 | Yes
B | [config not available] | | |
C | A* B C | 4 | 59 | Yes
Tablet 49ae66ad88f14c8cbbef641d7e220ce3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 58b12df3d325441687a7137795377032
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 73 | Yes
B | A B* C | 4 | 73 | Yes
C | [config not available] | | |
Tablet b3855f12777d4da2a45005df14af19d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [NONVOTER]
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 87a29ac48d5c4336a84fe85849b9f9ce
C = 58b12df3d325441687a7137795377032
D = 71fb1ea6da59487aaab9fc97a86fe8a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 5 | 73 | Yes
B | [config not available] | | |
C | A* B C D~ | 5 | 73 | Yes
D | A* B C D~ | 5 | 73 | Yes
Tablet 1fa13beb477d4e7f8cb2e23b40b9ad7f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = ca463f8d8c824dcbbef9d371959005a8
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 5 | 71 | Yes
B | A B* C | 5 | 71 | Yes
C | [config not available] | | |
Tablet ef83bf9db5aa432493376156741250c0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable [NONVOTER]
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = ca463f8d8c824dcbbef9d371959005a8
C = 1fc76a588ef94962b1098ef548930a7a
D = 87a29ac48d5c4336a84fe85849b9f9ce
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | A B* C | 8 | 76 | Yes
B | A B* C | 8 | 76 | Yes
C | [config not available] | | |
D | [config not available] | | |
Tablet 8a929c14ce3e4afe9a1b21410905db35 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [NONVOTER]
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 87a29ac48d5c4336a84fe85849b9f9ce
C = 58b12df3d325441687a7137795377032
D = 71fb1ea6da59487aaab9fc97a86fe8a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 6 | 77 | Yes
B | [config not available] | | |
C | A* B C D~ | 6 | 77 | Yes
D | A* B C D~ | 6 | 77 | Yes
Tablet 052ee510f96544cebd7d765d6564d1d9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = ca463f8d8c824dcbbef9d371959005a8
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 68 | Yes
B | A B* C | 4 | 68 | Yes
C | [config not available] | | |
Tablet 350c1957d7074c5d9ae07bccd54e589a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = ca463f8d8c824dcbbef9d371959005a8
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | 76 | Yes
B | A B* C | 6 | 76 | Yes
C | [config not available] | | |
Tablet 075d3725e1e94ad99828fc6782374d21 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = ca463f8d8c824dcbbef9d371959005a8
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 72 | Yes
B | A B* C | 4 | 72 | Yes
C | [config not available] | | |
Tablet ffec49d3e436496c9a2586660c496010 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = ca463f8d8c824dcbbef9d371959005a8
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 4 | 76 | Yes
B | A B* C | 4 | 76 | Yes
C | [config not available] | | |
Tablet 119e22dfdc5e45de98533476886dac4a of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [NONVOTER]
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 87a29ac48d5c4336a84fe85849b9f9ce
C = 58b12df3d325441687a7137795377032
D = 71fb1ea6da59487aaab9fc97a86fe8a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 4 | 69 | Yes
B | [config not available] | | |
C | A* B C D~ | 4 | 69 | Yes
D | A* B C D~ | 4 | 69 | Yes
Tablet b633bdcdedd548e89baf5a1d1907f2c3 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = ca463f8d8c824dcbbef9d371959005a8
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 5 | 69 | Yes
B | A B* C | 5 | 69 | Yes
C | [config not available] | | |
Tablet bc78439215134ac3b22d3c79f3904078 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [NONVOTER]
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 87a29ac48d5c4336a84fe85849b9f9ce
C = 58b12df3d325441687a7137795377032
D = 71fb1ea6da59487aaab9fc97a86fe8a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 2 | 71 | Yes
B | [config not available] | | |
C | A* B C D~ | 2 | 71 | Yes
D | A* B C D~ | 2 | 71 | Yes
Tablet 3f4c7140c09841448258f2633a5ee7db of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [LEADER]
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING
All reported replicas are:
A = 1fc76a588ef94962b1098ef548930a7a
B = 71fb1ea6da59487aaab9fc97a86fe8a5
C = ca463f8d8c824dcbbef9d371959005a8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 3 | 70 | Yes
C | A B* C | 3 | 70 | Yes
Tablet e4853a7585fb4cbf97550aa324804a6f of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING [LEADER]
87a29ac48d5c4336a84fe85849b9f9ce (127.2.59.4:41237): TS unavailable
58b12df3d325441687a7137795377032 (127.2.59.6:39375): RUNNING
71fb1ea6da59487aaab9fc97a86fe8a5 (127.2.59.5:44551): RUNNING [NONVOTER]
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = 87a29ac48d5c4336a84fe85849b9f9ce
C = 58b12df3d325441687a7137795377032
D = 71fb1ea6da59487aaab9fc97a86fe8a5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C D~ | | | Yes
A | A* B C D~ | 6 | 74 | Yes
B | [config not available] | | |
C | A* B C D~ | 6 | 74 | Yes
D | A B C D~ | 6 | 74 | Yes
Tablet 0feb41f93a3b44c68210b751a40a01e2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
8e70ad2eb3794816bfe677d38ddf729e (127.2.59.2:34389): RUNNING
ca463f8d8c824dcbbef9d371959005a8 (127.2.59.3:44853): RUNNING [LEADER]
1fc76a588ef94962b1098ef548930a7a (127.2.59.1:44619): TS unavailable
All reported replicas are:
A = 8e70ad2eb3794816bfe677d38ddf729e
B = ca463f8d8c824dcbbef9d371959005a8
C = 1fc76a588ef94962b1098ef548930a7a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 67 | Yes
B | A B* C | 3 | 67 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 4 | 0 | 16 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 6
First Quartile | 10
Median | 12
Third Quartile | 12
Maximum | 18
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 18
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 6
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 69
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.5:44551, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.2:34389, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.5:44551, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.5:44551, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.2:34389, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.5:44551, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.5:44551, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.3:44853, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.1:44619, tserver@127.2.59.6:39375, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
148d95f107fb4173ad7abf9b41a551d6 | 127.2.59.62:38965 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.2.59.20:41821 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+-------------+----------+----------------+-----------------
1fc76a588ef94962b1098ef548930a7a | 127.2.59.1:44619 | HEALTHY | <none> | 1 | 0
58b12df3d325441687a7137795377032 | 127.2.59.6:39375 | HEALTHY | <none> | 6 | 0
71fb1ea6da59487aaab9fc97a86fe8a5 | 127.2.59.5:44551 | HEALTHY | <none> | 2 | 0
8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | HEALTHY | <none> | 5 | 0
ca463f8d8c824dcbbef9d371959005a8 | 127.2.59.3:44853 | HEALTHY | <none> | 7 | 0
87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.59.4:41237: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.59.4:41237: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-----------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.59.1 | experimental | 127.2.59.1:44619
local_ip_for_outbound_sockets | 127.2.59.2 | experimental | 127.2.59.2:34389
local_ip_for_outbound_sockets | 127.2.59.3 | experimental | 127.2.59.3:44853
local_ip_for_outbound_sockets | 127.2.59.5 | experimental | 127.2.59.5:44551
local_ip_for_outbound_sockets | 127.2.59.6 | experimental | 127.2.59.6:39375
never_fsync | true | unsafe,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.59.1:44619
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.59.2:34389
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.59.3:44853
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.59.5:44551
server_dump_info_path | /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.59.6:39375
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-----------------------------------------------------------------------------
builtin_ntp_servers | 127.2.59.20:41821 | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
time_source | builtin | 127.2.59.1:44619, 127.2.59.6:39375, 127.2.59.5:44551, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.59.62:38965, tserver@127.2.59.6:39375, tserver@127.2.59.1:44619, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 9
Median | 12
Third Quartile | 12
Maximum | 20
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Big | 8e70ad2eb3794816bfe677d38ddf729e | 127.2.59.2:34389 | 20
Small | 87a29ac48d5c4336a84fe85849b9f9ce | 127.2.59.4:41237 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
E20240501 14:03:43.565423 2284 external_mini_cluster.cc:943] Process with UUID 87a29ac48d5c4336a84fe85849b9f9ce has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240501 14:03:44.706138 2284 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUTCidK/build/asan/bin/kudu pid 2386
I20240501 14:03:45.057204 2284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUTCidK/build/asan/bin/kudu with pid 2386
I20240501 14:03:45.110138 2284 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUTCidK/build/asan/bin/kudu pid 2517
I20240501 14:03:45.463073 2284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUTCidK/build/asan/bin/kudu with pid 2517
I20240501 14:03:45.516343 2284 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUTCidK/build/asan/bin/kudu pid 2647
I20240501 14:03:45.837081 2284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUTCidK/build/asan/bin/kudu with pid 2647
I20240501 14:03:45.887352 2284 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUTCidK/build/asan/bin/kudu pid 2907
I20240501 14:03:46.211995 2284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUTCidK/build/asan/bin/kudu with pid 2907
I20240501 14:03:46.263290 2284 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUTCidK/build/asan/bin/kudu pid 3037
I20240501 14:03:46.585747 2284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUTCidK/build/asan/bin/kudu with pid 3037
I20240501 14:03:46.637756 2284 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUTCidK/build/asan/bin/kudu pid 2296
I20240501 14:03:46.922329 2284 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUTCidK/build/asan/bin/kudu with pid 2296
2024-05-01T14:03:46Z chronyd exiting
I20240501 14:03:46.983004 2284 test_util.cc:175] -----------------------------------------------
I20240501 14:03:46.983139 2284 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUTCidK/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714572093596278-2284-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (133321 ms)
[----------] 1 test from RaftConsensusStressITest (133321 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (133321 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST
I20240501 14:03:46.985711 2284 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 4 messages since previous log ~58 seconds ago