[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RaftConsensusStressITest
[ RUN ] RaftConsensusStressITest.RemoveReplaceInCycle
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:16:31.876631 2846 test_util.cc:268] Using random seed: 947967465
I20240502 08:16:31.887840 2846 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:31.887925 2846 ts_itest-base.cc:113] --------------
I20240502 08:16:31.887979 2846 ts_itest-base.cc:114] 6 tablet servers
I20240502 08:16:31.888043 2846 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:16:31.888113 2846 ts_itest-base.cc:116] --------------
2024-05-02T08:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:31Z Disabled control of system clock
I20240502 08:16:31.909910 2846 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
/tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.199.190:41723
--webserver_interface=127.2.199.190
--webserver_port=0
--builtin_ntp_servers=127.2.199.148:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.199.190:41723
--minloglevel=2
--raft_prepare_replacement_before_eviction=true
--max_create_tablets_per_ts=20 with env {}
I20240502 08:16:32.291847 2846 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu as pid 2858
I20240502 08:16:32.292058 2846 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:32.380596 2846 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
/tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.199.129:0
--local_ip_for_outbound_sockets=127.2.199.129
--webserver_interface=127.2.199.129
--webserver_port=0
--tserver_master_addrs=127.2.199.190:41723
--builtin_ntp_servers=127.2.199.148:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 08:16:32.794277 2846 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu as pid 2949
I20240502 08:16:32.794514 2846 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:32.798031 2846 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
/tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.199.130:0
--local_ip_for_outbound_sockets=127.2.199.130
--webserver_interface=127.2.199.130
--webserver_port=0
--tserver_master_addrs=127.2.199.190:41723
--builtin_ntp_servers=127.2.199.148:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 08:16:33.222463 2846 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu as pid 3080
I20240502 08:16:33.222643 2846 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:33.225948 2846 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
/tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.199.131:0
--local_ip_for_outbound_sockets=127.2.199.131
--webserver_interface=127.2.199.131
--webserver_port=0
--tserver_master_addrs=127.2.199.190:41723
--builtin_ntp_servers=127.2.199.148:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 08:16:33.640364 2846 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu as pid 3211
I20240502 08:16:33.640588 2846 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:33.643954 2846 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
/tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.199.132:0
--local_ip_for_outbound_sockets=127.2.199.132
--webserver_interface=127.2.199.132
--webserver_port=0
--tserver_master_addrs=127.2.199.190:41723
--builtin_ntp_servers=127.2.199.148:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 08:16:34.079906 2846 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu as pid 3343
I20240502 08:16:34.080089 2846 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:34.083434 2846 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
/tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.199.133:0
--local_ip_for_outbound_sockets=127.2.199.133
--webserver_interface=127.2.199.133
--webserver_port=0
--tserver_master_addrs=127.2.199.190:41723
--builtin_ntp_servers=127.2.199.148:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 08:16:34.517787 2846 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu as pid 3474
I20240502 08:16:34.517948 2846 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:34.521389 2846 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
/tmp/dist-test-taskPEUtcp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.199.134:0
--local_ip_for_outbound_sockets=127.2.199.134
--webserver_interface=127.2.199.134
--webserver_port=0
--tserver_master_addrs=127.2.199.190:41723
--builtin_ntp_servers=127.2.199.148:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--minloglevel=2
--tablet_copy_early_session_timeout_prob=0.125
--raft_prepare_replacement_before_eviction=true
--follower_unavailable_considered_failed_sec=2
--consensus_rpc_timeout_ms=2000
--heartbeat_interval_ms=500
--raft_heartbeat_interval_ms=200 with env {}
I20240502 08:16:34.947623 2846 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu as pid 3604
I20240502 08:16:34.947775 2846 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:16:34.964078 2846 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 08:16:35.162937 2846 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 08:16:35.165359 2846 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cf7697cf028248319a2d660fb587d992 to finish bootstrapping
I20240502 08:16:35.174211 2846 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fb92dfdd45d54697b3ddd26ceab365c7 to finish bootstrapping
I20240502 08:16:35.181623 2846 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5832f1788a704a3c9f2a828a1fb63e5c to finish bootstrapping
I20240502 08:16:35.189620 2846 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 604cd62787684cad881818ed0e753c75 to finish bootstrapping
I20240502 08:16:35.196672 2846 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3228b771d5e64608ac57507b6a86efa4 to finish bootstrapping
I20240502 08:16:35.203471 2846 ts_itest-base.cc:233] Waiting for 0 tablets on tserver caca2ab7b432408fb2be68e97ed7dd55 to finish bootstrapping
I20240502 08:16:35.211745 2846 test_util.cc:268] Using random seed: 951302586
W20240502 08:16:50.230257 4331 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:16:47.003992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.199.133:40437 (local address 127.0.0.1:37244)
0502 08:16:47.004706 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0502 08:16:47.004728 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0502 08:16:47.004872 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:16:50.228734 (+3223862us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.199.133:40437: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":591}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | HEALTHY | <none> | 4 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 1 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 3 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 5 | 0
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.133:40437: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.133:40437: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.132 | experimental | 127.2.199.132:44475
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.199.132:44475
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.134:37187, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
Tablet 63245f12091949d9903133cf6b70edc4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable [LEADER]
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
Tablet acd55337b543412f896b2605d634cdb8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable [LEADER]
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1869834b0fe44b128e846a8e4804f8d8 of table 'RemoveReplaceInCycle' is conflicted: 3 replicas' active configs disagree with the leader master's
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
All reported replicas are:
A = cf7697cf028248319a2d660fb587d992
B = fb92dfdd45d54697b3ddd26ceab365c7
C = 5832f1788a704a3c9f2a828a1fb63e5c
D = 604cd62787684cad881818ed0e753c75
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C D | | | Yes
A | A B* C D | 1 | 45 | Yes
B | A B D | 2 | 46 | No
C | A B C D~ | 2 | 44 | Yes
D | A B* D | 1 | 46 | No
Tablet a366bff2b70a440fb9b1ae8b964d7641 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING [LEADER]
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 9fdac8e0a0bb407fa4aedaf626ae23a9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING [LEADER]
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet d783e40572d845269f048e2391c730b5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING [LEADER]
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 8cfb39606dec4a128b0b36b2ef12e67c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable [LEADER]
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 3afaad2e6c3144e2acce26011ac950b7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING [LEADER]
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 0d818dcd97d1400790d359a2e6e06535 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable [LEADER]
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 977844131ab74ab5a5cd3bafcd8b3ef8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable [LEADER]
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 0d6e947143c842d7ac4dcc99839d8ce4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): not running
State: STOPPED
Data state: TABLET_DATA_TOMBSTONED
Last status: No bootstrap required, opened a new log
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
All reported replicas are:
A = cf7697cf028248319a2d660fb587d992
B = fb92dfdd45d54697b3ddd26ceab365c7
C = 5832f1788a704a3c9f2a828a1fb63e5c
D = 604cd62787684cad881818ed0e753c75
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------+--------------+--------------+------------
master | A B* C D | | | Yes
A | A B C D | 2 | 41 | Yes
B | B* C D | 2 | 43 | Yes
C | B* C D | 2 | 43 | Yes
D | B* C D | 2 | 43 | Yes
Tablet a8e04338c98641688b56c71b3eb620d4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable [LEADER]
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = caca2ab7b432408fb2be68e97ed7dd55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet e55ca4bfa4a7458b9bd2cc728bcd7fe4 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
All reported replicas are:
A = 3228b771d5e64608ac57507b6a86efa4
B = 604cd62787684cad881818ed0e753c75
C = fb92dfdd45d54697b3ddd26ceab365c7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | CONSENSUS_MISMATCH | 20 | 8 | 0 | 11 | 1
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 10
Median | 11
Third Quartile | 12
Maximum | 19
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 19
Small | 5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 65
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
W20240502 08:16:56.888969 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:16:53.699084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.199.131:37801 (local address 127.0.0.1:48344)
0502 08:16:53.712627 (+ 13543us) negotiation.cc:107] Waiting for socket to connect
0502 08:16:53.712644 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 08:16:53.712755 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:16:56.888651 (+3175896us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.199.131:37801: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":13423,"thread_start_us":30036,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 0 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | HEALTHY | <none> | 8 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 4 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 3 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 6 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.131:37801: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.131:37801: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.132 | experimental | 127.2.199.132:44475
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.199.132:44475
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.132:44475, and 3 other server(s)
Tablet Summary
Tablet e55ca4bfa4a7458b9bd2cc728bcd7fe4 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING [LEADER]
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = fb92dfdd45d54697b3ddd26ceab365c7
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 5 | Yes
B | A* B C | 2 | 5 | Yes
C | [config not available] | | |
Tablet 540d4216360e428785c5eb8ef48951ca of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING [LEADER]
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = cf7697cf028248319a2d660fb587d992
B = fb92dfdd45d54697b3ddd26ceab365c7
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | A* B C | 2 | -1 | Yes
C | [config not available] | | |
Tablet 0d6e947143c842d7ac4dcc99839d8ce4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 5832f1788a704a3c9f2a828a1fb63e5c
C = 604cd62787684cad881818ed0e753c75
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 43 | Yes
B | [config not available] | | |
C | A* B C | 2 | 43 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 18 | 0 | 2 | 0
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 3
Median | 12
Third Quartile | 18
Maximum | 19
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 0 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | HEALTHY | <none> | 8 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 4 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 3 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.132 | experimental | 127.2.199.132:44475
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.199.132:44475
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 12
Third Quartile | 18
Maximum | 20
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:17:01.055107 2846 raft_consensus_stress-itest.cc:258] completed iteration 0
W20240502 08:17:10.708966 5379 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:17:07.487259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.199.131:37801 (local address 127.0.0.1:39238)
0502 08:17:07.500499 (+ 13240us) negotiation.cc:107] Waiting for socket to connect
0502 08:17:07.500516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 08:17:07.500626 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:17:10.708676 (+3208050us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.199.131:37801: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":13080,"thread_start_us":9931,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | HEALTHY | <none> | 8 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 0 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 9 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.131:37801: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.131:37801: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.132 | experimental | 127.2.199.132:44475
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.199.132:44475
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.132:44475, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
Tablet 63245f12091949d9903133cf6b70edc4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 53 | Yes
B | A* B C | 2 | 53 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 19 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 20
Third Quartile | 20
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | HEALTHY | <none> | 9 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 1 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 7 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.132 | experimental | 127.2.199.132:44475
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.199.132:44475
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 20
Third Quartile | 20
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:17:14.524963 2846 raft_consensus_stress-itest.cc:258] completed iteration 1
W20240502 08:17:23.508942 6015 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:17:20.375351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.199.133:40437 (local address 127.0.0.1:35796)
0502 08:17:20.375555 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0502 08:17:20.375575 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0502 08:17:20.375699 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:17:23.508724 (+3133025us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.199.133:40437: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":63}
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | HEALTHY | <none> | 12 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 0 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 1 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 7 | 0
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.133:40437: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.133:40437: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.132 | experimental | 127.2.199.132:44475
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.199.132:44475
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
Tablet fb4d9384eca740c3b2047268e8c7ad88 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING [LEADER]
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 59 | Yes
B | A B* C | 3 | 59 | Yes
C | [config not available] | | |
Tablet fc92e0f6cd974d52bd4aa08d120c4c32 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING [LEADER]
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 56 | Yes
B | A B* C | 2 | 56 | Yes
C | [config not available] | | |
Tablet acd55337b543412f896b2605d634cdb8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable [LEADER]
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = 3228b771d5e64608ac57507b6a86efa4
C = cf7697cf028248319a2d660fb587d992
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 58 | Yes
B | [config not available] | | |
C | A B* C | 3 | 58 | Yes
Tablet 1869834b0fe44b128e846a8e4804f8d8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING [LEADER]
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 62 | Yes
B | A B* C | 2 | 62 | Yes
C | [config not available] | | |
Tablet f9504b3d6ca941898a557c8bcd4372a2 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 61 | Yes
B | A* B C | 3 | 61 | Yes
C | [config not available] | | |
Tablet 540d4216360e428785c5eb8ef48951ca of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 5 | 59 | Yes
B | A* B C | 5 | 59 | Yes
C | [config not available] | | |
Tablet 3948fee0eca54836875b59a2ea5e5cd8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 61 | Yes
B | A* B C | 3 | 61 | Yes
C | [config not available] | | |
Tablet 71f51870fd5e423ab21887a81fb994a0 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 59 | Yes
B | A* B C | 1 | 59 | Yes
C | [config not available] | | |
Tablet 6569ae0a97534ddf8c1bc4d8a60fb488 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 56 | Yes
B | A* B C | 1 | 56 | Yes
C | [config not available] | | |
Tablet 295fc758a4f64b39adf05556a91f7883 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 57 | Yes
B | A* B C | 3 | 57 | Yes
C | [config not available] | | |
Tablet 0d6e947143c842d7ac4dcc99839d8ce4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = 604cd62787684cad881818ed0e753c75
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 58 | Yes
B | A* B C | 2 | 58 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 9 | 0 | 11 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 11
Third Quartile | 20
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 0 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | HEALTHY | <none> | 11 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 0 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 3 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 7 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.132 | experimental | 127.2.199.132:44475
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.199.132:44475
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 12
Third Quartile | 21
Maximum | 21
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:17:28.050102 2846 raft_consensus_stress-itest.cc:258] completed iteration 2
W20240502 08:17:37.592876 6791 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:17:34.383986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.199.131:37801 (local address 127.0.0.1:57792)
0502 08:17:34.396837 (+ 12851us) negotiation.cc:107] Waiting for socket to connect
0502 08:17:34.396858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0502 08:17:34.397016 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:17:37.592662 (+3195646us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.2.199.131:37801: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":12699,"thread_start_us":12567,"threads_started":1}
F20240502 08:17:37.968750 6890 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0d6e947143c842d7ac4dcc99839d8ce4.queue_time_us, tablet-download-pool-0d6e947143c842d7ac4dcc99839d8ce4.queue_time_us) (tablet-download-pool-0d6e947143c842d7ac4dcc99839d8ce4.run_cpu_time_us, tablet-download-pool-0d6e947143c842d7ac4dcc99839d8ce4.run_cpu_time_us) (tablet-download-pool-0d6e947143c842d7ac4dcc99839d8ce4.run_wall_time_us, tablet-download-pool-0d6e947143c842d7ac4dcc99839d8ce4.run_wall_time_us) (tablet-download-pool-0d818dcd97d1400790d359a2e6e06535.queue_time_us, tablet-download-pool-0d818dcd97d1400790d359a2e6e06535.queue_time_us) (tablet-download-pool-0d818dcd97d1400790d359a2e6e06535.run_cpu_time_us, tablet-download-pool-0d818dcd97d1400790d359a2e6e06535.run_cpu_time_us) (tablet-download-pool-0d818dcd97d1400790d359a2e6e06535.run_wall_time_us, tablet-download-pool-0d818dcd97d1400790d359a2e6e06535.run_wall_time_us) (tablet-download-pool-1869834b0fe44b128e846a8e4804f8d8.queue_time_us, tablet-download-pool-1869834b0fe44b128e846a8e4804f8d8.queue_time_us) (tablet-download-pool-1869834b0fe44b128e846a8e4804f8d8.run_cpu_time_us, tablet-download-pool-1869834b0fe44b128e846a8e4804f8d8.run_cpu_time_us) (tablet-download-pool-1869834b0fe44b128e846a8e4804f8d8.run_wall_time_us, tablet-download-pool-1869834b0fe44b128e846a8e4804f8d8.run_wall_time_us) (tablet-download-pool-295fc758a4f64b39adf05556a91f7883.queue_time_us, tablet-download-pool-295fc758a4f64b39adf05556a91f7883.queue_time_us) (tablet-download-pool-295fc758a4f64b39adf05556a91f7883.run_cpu_time_us, tablet-download-pool-295fc758a4f64b39adf05556a91f7883.run_cpu_time_us) (tablet-download-pool-295fc758a4f64b39adf05556a91f7883.run_wall_time_us, tablet-download-pool-295fc758a4f64b39adf05556a91f7883.run_wall_time_us) (tablet-download-pool-3948fee0eca54836875b59a2ea5e5cd8.queue_time_us, tablet-download-pool-3948fee0eca54836875b59a2ea5e5cd8.queue_time_us) (tablet-download-pool-3948fee0eca54836875b59a2ea5e5cd8.run_cpu_time_us, tablet-download-pool-3948fee0eca54836875b59a2ea5e5cd8.run_cpu_time_us) (tablet-download-pool-3948fee0eca54836875b59a2ea5e5cd8.run_wall_time_us, tablet-download-pool-3948fee0eca54836875b59a2ea5e5cd8.run_wall_time_us) (tablet-download-pool-3afaad2e6c3144e2acce26011ac950b7.queue_time_us, tablet-download-pool-3afaad2e6c3144e2acce26011ac950b7.queue_time_us) (tablet-download-pool-3afaad2e6c3144e2acce26011ac950b7.run_cpu_time_us, tablet-download-pool-3afaad2e6c3144e2acce26011ac950b7.run_cpu_time_us) (tablet-download-pool-3afaad2e6c3144e2acce26011ac950b7.run_wall_time_us, tablet-download-pool-3afaad2e6c3144e2acce26011ac950b7.run_wall_time_us) (tablet-download-pool-540d4216360e428785c5eb8ef48951ca.queue_time_us, tablet-download-pool-540d4216360e428785c5eb8ef48951ca.queue_time_us) (tablet-download-pool-540d4216360e428785c5eb8ef48951ca.run_cpu_time_us, tablet-download-pool-540d4216360e428785c5eb8ef48951ca.run_cpu_time_us) (tablet-download-pool-540d4216360e428785c5eb8ef48951ca.run_wall_time_us, tablet-download-pool-540d4216360e428785c5eb8ef48951ca.run_wall_time_us) (tablet-download-pool-6569ae0a97534ddf8c1bc4d8a60fb488.queue_time_us, tablet-download-pool-6569ae0a97534ddf8c1bc4d8a60fb488.queue_time_us) (tablet-download-pool-6569ae0a97534ddf8c1bc4d8a60fb488.run_cpu_time_us, tablet-download-pool-6569ae0a97534ddf8c1bc4d8a60fb488.run_cpu_time_us) (tablet-download-pool-6569ae0a97534ddf8c1bc4d8a60fb488.run_wall_time_us, tablet-download-pool-6569ae0a97534ddf8c1bc4d8a60fb488.run_wall_time_us) (tablet-download-pool-71f51870fd5e423ab21887a81fb994a0.queue_time_us, tablet-download-pool-71f51870fd5e423ab21887a81fb994a0.queue_time_us) (tablet-download-pool-71f51870fd5e423ab21887a81fb994a0.run_cpu_time_us, tablet-download-pool-71f51870fd5e423ab21887a81fb994a0.run_cpu_time_us) (tablet-download-pool-71f51870fd5e423ab21887a81fb994a0.run_wall_time_us, tablet-download-pool-71f51870fd5e423ab21887a81fb994a0.run_wall_time_us) (tablet-download-pool-9fdac8e0a0bb407fa4aedaf626ae23a9.queue_time_us, tablet-download-pool-9fdac8e0a0bb407fa4aedaf626ae23a9.queue_time_us) (tablet-download-pool-9fdac8e0a0bb407fa4aedaf626ae23a9.run_cpu_time_us, tablet-download-pool-9fdac8e0a0bb407fa4aedaf626ae23a9.run_cpu_time_us) (tablet-download-pool-9fdac8e0a0bb407fa4aedaf626ae23a9.run_wall_time_us, tablet-download-pool-9fdac8e0a0bb407fa4aedaf626ae23a9.run_wall_time_us) (tablet-download-pool-acd55337b543412f896b2605d634cdb8.queue_time_us, tablet-download-pool-acd55337b543412f896b2605d634cdb8.queue_time_us) (tablet-download-pool-d783e40572d845269f048e2391c730b5.queue_time_us, tablet-download-pool-d783e40572d845269f048e2391c730b5.queue_time_us) (tablet-download-pool-d783e40572d845269f048e2391c730b5.run_cpu_time_us, tablet-download-pool-d783e40572d845269f048e2391c730b5.run_cpu_time_us) (tablet-download-pool-d783e40572d845269f048e2391c730b5.run_wall_time_us, tablet-download-pool-d783e40572d845269f048e2391c730b5.run_wall_time_us) (tablet-download-pool-f9504b3d6ca941898a557c8bcd4372a2.queue_time_us, tablet-download-pool-f9504b3d6ca941898a557c8bcd4372a2.queue_time_us) (tablet-download-pool-f9504b3d6ca941898a557c8bcd4372a2.run_cpu_time_us, tablet-download-pool-f9504b3d6ca941898a557c8bcd4372a2.run_cpu_time_us) (tablet-download-pool-f9504b3d6ca941898a557c8bcd4372a2.run_wall_time_us, tablet-download-pool-f9504b3d6ca941898a557c8bcd4372a2.run_wall_time_us) (tablet-download-pool-fb4d9384eca740c3b2047268e8c7ad88.queue_time_us, tablet-download-pool-fb4d9384eca740c3b2047268e8c7ad88.queue_time_us) (tablet-download-pool-fb4d9384eca740c3b2047268e8c7ad88.run_cpu_time_us, tablet-download-pool-fb4d9384eca740c3b2047268e8c7ad88.run_cpu_time_us) (tablet-download-pool-fb4d9384eca740c3b2047268e8c7ad88.run_wall_time_us, tablet-download-pool-fb4d9384eca740c3b2047268e8c7ad88.run_wall_time_us) (tablet-download-pool-fc92e0f6cd974d52bd4aa08d120c4c32.queue_time_us, tablet-download-pool-fc92e0f6cd974d52bd4aa08d120c4c32.queue_time_us) (tablet-download-pool-fc92e0f6cd974d52bd4aa08d120c4c32.run_cpu_time_us, tablet-download-pool-fc92e0f6cd974d52bd4aa08d120c4c32.run_cpu_time_us) (tablet-download-pool-fc92e0f6cd974d52bd4aa08d120c4c32.run_wall_time_us, tablet-download-pool-fc92e0f6cd974d52bd4aa08d120c4c32.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7ff096e0adcd google::LogMessage::Fail() at ??:0
@ 0x7ff096e0eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff096e0a7cc google::LogMessage::Flush() at ??:0
@ 0x7ff096e0bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff099b4b742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7ff099b17f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7ff099b11cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7ff0a8f648de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7ff0a8f7cdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7ff0a90a7480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7ff0a90dfa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7ff0a90c2981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7ff0a90c270e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ff099b1bb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff099b1f19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff099b1f019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ff099afa01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff09ce7f6db start_thread at ??:0
@ 0x7ff095cbd71f clone at ??:0
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 0 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 2 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 6 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 12 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.131:37801: Timed out: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.131:37801: received 0 of 4 requested bytes (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.132 | experimental | 127.2.199.132:44475
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.2.199.132:44475
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | all 6 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.132:44475, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
Tablet e55ca4bfa4a7458b9bd2cc728bcd7fe4 of table 'TestTable' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 15 | Yes
B | A* B C | 3 | 15 | Yes
C | [config not available] | | |
Tablet 63245f12091949d9903133cf6b70edc4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING [LEADER]
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 68 | Yes
B | A B* C | 3 | 68 | Yes
C | [config not available] | | |
Tablet acd55337b543412f896b2605d634cdb8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING [LEADER]
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = cf7697cf028248319a2d660fb587d992
B = fb92dfdd45d54697b3ddd26ceab365c7
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 69 | Yes
B | A* B C | 4 | 69 | Yes
C | [config not available] | | |
Tablet 9fdac8e0a0bb407fa4aedaf626ae23a9 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 5 | 68 | Yes
B | A* B C | 5 | 68 | Yes
C | [config not available] | | |
Tablet d783e40572d845269f048e2391c730b5 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING [LEADER]
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 68 | Yes
B | A B* C | 2 | 68 | Yes
C | [config not available] | | |
Tablet 8cfb39606dec4a128b0b36b2ef12e67c of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING [LEADER]
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 67 | Yes
B | A B* C | 3 | 67 | Yes
C | [config not available] | | |
Tablet 3afaad2e6c3144e2acce26011ac950b7 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING [LEADER]
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 68 | Yes
B | A B* C | 2 | 68 | Yes
C | [config not available] | | |
Tablet 6569ae0a97534ddf8c1bc4d8a60fb488 of table 'RemoveReplaceInCycle' is conflicted: 1 replicas' active configs disagree with the leader master's
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
caca2ab7b432408fb2be68e97ed7dd55 (127.2.199.134:37187): RUNNING
3228b771d5e64608ac57507b6a86efa4 (127.2.199.133:40437): RUNNING
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = caca2ab7b432408fb2be68e97ed7dd55
C = 3228b771d5e64608ac57507b6a86efa4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A B C | 2 | 66 | Yes
B | A* B C | 1 | 66 | Yes
C | A* B C | 1 | 66 | Yes
Tablet 0d818dcd97d1400790d359a2e6e06535 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING [LEADER]
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | 64 | Yes
B | A B* C | 3 | 64 | Yes
C | [config not available] | | |
Tablet 977844131ab74ab5a5cd3bafcd8b3ef8 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | 67 | Yes
B | A* B C | 4 | 67 | Yes
C | [config not available] | | |
Tablet a8e04338c98641688b56c71b3eb620d4 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): TS unavailable
All reported replicas are:
A = fb92dfdd45d54697b3ddd26ceab365c7
B = cf7697cf028248319a2d660fb587d992
C = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 3 | 64 | Yes
B | A* B C | 3 | 64 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+--------------------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | CONSENSUS_MISMATCH | 20 | 10 | 0 | 9 | 1
TestTable | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 9
Median | 10
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 2 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
Tablet a366bff2b70a440fb9b1ae8b964d7641 of table 'RemoveReplaceInCycle' is under-replicated: 1 replica(s) not RUNNING
604cd62787684cad881818ed0e753c75 (127.2.199.132:44475): TS unavailable
fb92dfdd45d54697b3ddd26ceab365c7 (127.2.199.130:43785): RUNNING [LEADER]
cf7697cf028248319a2d660fb587d992 (127.2.199.129:41183): RUNNING
5832f1788a704a3c9f2a828a1fb63e5c (127.2.199.131:37801): missing [NONVOTER]
All reported replicas are:
A = 604cd62787684cad881818ed0e753c75
B = fb92dfdd45d54697b3ddd26ceab365c7
C = cf7697cf028248319a2d660fb587d992
D = 5832f1788a704a3c9f2a828a1fb63e5c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C D~ | | | Yes
A | [config not available] | | |
B | A B* C D~ | 3 | 66 | Yes
C | A B* C D~ | 3 | 66 | Yes
D | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
RemoveReplaceInCycle | 3 | UNDER_REPLICATED | 20 | 19 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 64
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.130:43785, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.134:37187, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.134:37187, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.134:37187, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.134:37187, tserver@127.2.199.130:43785, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.131:37801, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.129:41183, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.131:37801, tserver@127.2.199.133:40437, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bfa7ed72d1f04846bb61f6d0ec2c3641 | 127.2.199.190:41723 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.2.199.148:34839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
3228b771d5e64608ac57507b6a86efa4 | 127.2.199.133:40437 | HEALTHY | <none> | 1 | 0
5832f1788a704a3c9f2a828a1fb63e5c | 127.2.199.131:37801 | HEALTHY | <none> | 0 | 0
caca2ab7b432408fb2be68e97ed7dd55 | 127.2.199.134:37187 | HEALTHY | <none> | 3 | 0
cf7697cf028248319a2d660fb587d992 | 127.2.199.129:41183 | HEALTHY | <none> | 7 | 0
fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | HEALTHY | <none> | 10 | 0
604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 6
Error from 127.2.199.132:44475: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.2.199.132:44475: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+--------------------------------------------------------------------------------------
ipki_server_key_size | 768 | experimental | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
local_ip_for_outbound_sockets | 127.2.199.129 | experimental | 127.2.199.129:41183
local_ip_for_outbound_sockets | 127.2.199.130 | experimental | 127.2.199.130:43785
local_ip_for_outbound_sockets | 127.2.199.131 | experimental | 127.2.199.131:37801
local_ip_for_outbound_sockets | 127.2.199.133 | experimental | 127.2.199.133:40437
local_ip_for_outbound_sockets | 127.2.199.134 | experimental | 127.2.199.134:37187
never_fsync | true | unsafe,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
openssl_security_level_override | 0 | unsafe,hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
raft_prepare_replacement_before_eviction | true | experimental,advanced | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
rpc_server_allow_ephemeral_ports | true | unsafe | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_format | pb | hidden | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.199.129:41183
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.199.130:43785
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.199.131:37801
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.2.199.133:40437
server_dump_info_path | /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.2.199.134:37187
tablet_copy_early_session_timeout_prob | 0.125 | unsafe,runtime | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+--------------------------------------------------------------------------------------
builtin_ntp_servers | 127.2.199.148:34839 | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
time_source | builtin | 127.2.199.133:40437, 127.2.199.131:37801, 127.2.199.134:37187, and 2 other server(s)
Version Summary
Version | Servers
-----------------+-------------------------------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.2.199.190:41723, tserver@127.2.199.133:40437, tserver@127.2.199.134:37187, and 3 other server(s)
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------------+----+---------+---------------+---------+------------+------------------+-------------
RemoveReplaceInCycle | 3 | HEALTHY | 20 | 20 | 0 | 0 | 0
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 8
Median | 11
Third Quartile | 12
Maximum | 21
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Big | fb92dfdd45d54697b3ddd26ceab365c7 | 127.2.199.130:43785 | 21
Small | 604cd62787684cad881818ed0e753c75 | 127.2.199.132:44475 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 2
Tablets | 21
Replicas | 63
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 6 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 6 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 6 had errors
FAILED
E20240502 08:18:20.180902 2846 external_mini_cluster.cc:943] Process with UUID 604cd62787684cad881818ed0e753c75 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_stress-itest.cc:263: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 08:18:20.538082 2846 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu pid 2949
I20240502 08:18:20.874194 2846 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu with pid 2949
I20240502 08:18:20.925216 2846 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu pid 3080
I20240502 08:18:21.289875 2846 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu with pid 3080
I20240502 08:18:21.339102 2846 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu pid 3211
I20240502 08:18:21.661118 2846 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu with pid 3211
I20240502 08:18:21.701567 2846 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu pid 3474
I20240502 08:18:22.013579 2846 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu with pid 3474
I20240502 08:18:22.056216 2846 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu pid 3604
I20240502 08:18:22.388332 2846 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu with pid 3604
I20240502 08:18:22.431246 2846 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu pid 2858
I20240502 08:18:22.705453 2846 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPEUtcp/build/asan/bin/kudu with pid 2858
2024-05-02T08:18:22Z chronyd exiting
I20240502 08:18:22.759336 2846 test_util.cc:175] -----------------------------------------------
I20240502 08:18:22.759426 2846 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPEUtcp/test-tmp/raft_consensus_stress-itest.0.RaftConsensusStressITest.RemoveReplaceInCycle.1714637791807215-2846-0
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle (110887 ms)
[----------] 1 test from RaftConsensusStressITest (110887 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (110887 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusStressITest.RemoveReplaceInCycle
1 FAILED TEST